Seven failure modes in a live alarm subsystem, each with the arithmetic that exposes it. From 7,839 rows on the board to 1,175. From 17 unrendered Critical alarms to 32 rendered. From a suppression rate reported as 14.2% to a measured 80.9%. An analysis of what an alarm system is actually doing to the attention of the people who watch it.
Contents
- What an alarm is supposed to be
- The board that deleted its own alarms
- Load, suppression, and the arithmetic of an operator's burden
- Priority resolution, and who is allowed to decide
- States are not conditions
- Key stability, latching, and why acknowledgement can stop meaning anything
- The monitor that stops
- Ten questions for your own alarm system
- What RenewOps does about this
Read offline
The complete reference is on this page. The PDF is for circulation inside your organization.
Download the PDFChapter one
What an alarm is supposed to be
An alarm is a claim about the plant that obliges a person to do something. Everything that does not meet that definition is a different kind of object wearing an alarm's clothes.
The definition has three parts, and each maps to a failure class that occupies a chapter of this document.
| Part | Means | Fails as |
|---|---|---|
| A claim about the plant | The condition is real and currently true. Not a status change, not a heartbeat, not the poller's opinion. | State events on the board (Ch. 5) |
| That obliges action | There is something a person should do. If the answer is "nothing," the object is information, and it belongs in a log. | Flood and suppression (Ch. 3) |
| By a person | A specific person, with a response, whose action is recorded against their identity and can be reconstructed. | Attribution failure (Ch. 6, and EC-WP-1102) |
The four objects that get confused with alarms
▸ State events. "Sunrise: Daylight Hours." A true statement about the world requiring nothing of anybody. On one assessed board this single event had latched 2,092 times.
▸ Conditions. A real state that persists and clears on its own — an inverter work-state transition with a 97.4% self-clear rate and a median duration of 0.56 minutes. Real, and not an alarm.
▸ Derived observations. "Live production 14% below forecast so far today." A computed judgement, useful, and not a plant condition.
▸ Diagnostics. Instrument health. Important, and addressed to a different person on a different timescale.
Each of the four is legitimate. Placing any of them on an operator's alarm board is what produces the flood, and the flood is what makes the board unreadable, and an unreadable board is how a trip-coil supervision failure sits at High priority for twenty-six days.
The measurement that settles the argument
Whether an object belongs on the board is not a matter of opinion. It is measurable, from the object's own history, with two numbers:
Calculation 1 · Board eligibility
-- Self-clear rate and median duration decide the class. SELECT event_name, COUNT(*) AS occurrences, COUNT(*) FILTER (WHERE cleared_without_action) AS self_cleared, ROUND(100.0 * COUNT(*) FILTER (WHERE cleared_without_action) / NULLIF(COUNT(*),0), 1) AS self_clear_pct, PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY EXTRACT(epoch FROM (cleared_at - raised_at))/60) AS median_minutes, COUNT(*) FILTER (WHERE state = 'active') AS active_now FROM alarm_history WHERE raised_at > now() - interval '90 days' GROUP BY event_name ORDER BY occurrences DESC;
Reading it. A high self-clear rate with a short median duration is transient chatter: it resolves before a human could act, so a human acting on it is impossible by construction. A low self-clear rate with a long median duration is a genuine standing condition. On one assessed catalogue, a trip-coil supervision code returned a self-clear rate of 0.0% with a median duration of 26 days — the signature of a real defect nobody was closing — while it sat at High priority beneath a flood of transients.
The rule this yields. An object belongs on the operator board only if a person can act on it before it resolves itself, and only if the action is one the operator is actually authorised to take. Everything else belongs in a log, a report, or a different person's queue — and moving it there is not suppression, it is filing.
Chapter two
The board that deleted its own alarms
Every alarm the platform raised itself was destroyed within thirty seconds of being raised, for months, and stamped as though the plant had returned to normal.
The defect
An ingest sweep reconciled the local alarm store against the array of active keys handed to it by the upstream monitoring platform on each cycle. Simplified, the predicate was:
-- The sweep, as it stood. UPDATE alarms SET active_upstream = false, state = CASE WHEN state = 'acked' THEN 'archived' ELSE 'auto_resolved' END, archived_at = now(), archive_reason = 'cleared_at_source' WHERE state IN ('active_unacked','acked') AND active_upstream = true AND last_seen_at < now();
Read the predicate carefully. It does not say this alarm cleared at source. It says this alarm is not in the array I was just handed. Those are different claims, and for one category of alarm they are catastrophically different: anything the platform computes itself is, by construction, never in that array.
What it cost
| Alarm generator | Rows ever raised | On the board | Closed as "cleared at source" |
|---|---|---|---|
| Voltage band excursion | 575 | 0 | 548 |
| Voltage schedule deviation | 262 | 0 | 246 |
| Regulation state change | 5 | 0 | 5 |
| Storage state-of-charge low | 1 | 0 | 1 |
| Total | 843 | 0 | 800 |
Not one of the 843 had ever survived a single thirty-second ingest cycle. The voltage-band and voltage-schedule alarms are the ones that matter most here: they are the compliance-relevant ones, in the standard family governing operation to a voltage schedule. 794 of them were raised and destroyed, each stamped with a reason asserting the plant had returned to normal. No operator ever saw one.
Ours. This defect was in RenewOps. It is published here for the same reason it was published in the assessment report: the mechanism is general, it will exist in any system that reconciles a local store against an upstream list, and a vendor who describes only other people's defects is describing a sales document rather than a method.
How it was found — and why it was nearly not
Not by an alarm about alarms. By an assertion. A storage state-of-charge monitor raised a low-charge alarm at 7.81% against a documented latch threshold of 10%. Thirty seconds later the row read auto_resolved. The monitor was correct; the sweep had eaten its output.
The general point is worth stating plainly, because it generalises well beyond alarms: the only reason this was findable is that something wrote down what it expected to happen and then checked. Without the documented threshold, the disappearance would have read as normal behaviour.
The fix, and the two second-order problems it exposed
The repair itself is one clause: derive an origin on the alarm row — a key carrying the upstream event-start timestamp is upstream, anything else was minted locally — and scope the sweep AND origin = 'upstream'. Two consequences surfaced immediately, and both are instructive.
Consequence one — unstable keys
Three of the four local generators embedded epoch milliseconds in the alarm key, so every evaluation minted a fresh row. That was harmless while the sweep deleted them all. The moment they survived, it became unbounded growth — and worse, it made acknowledgement meaningless, because the next evaluation would create a row nobody had acknowledged.
A key is not an identifier if it changes when the underlying condition does not. This is covered properly in Chapter 6.
Consequence two — the sweep had been hiding dead monitors
The sweep was quietly cleaning up after evaluators that had stopped. Remove it and a stopped monitor's last alarm stands forever, asserting a condition nobody is checking. In one observed instance a storage monitor stopped at 07:50; at 11:29 the board still showed a site at "6.7%, at or below 10%" while the plant actually read 19.86%.
The correct answer is not to reinstate the deletion. It is a heartbeat, and a per-alarm freshness label that marks an unconfirmed alarm as unconfirmed rather than clearing it. That is Chapter 7.
The design rule
A reconciliation predicate must state the claim it is actually making.
If the predicate is "not present in the upstream list," the reason code written must be not present upstream — not cleared at source. The two are different facts and the second is unsupported by the query that produced it.
Then: scope every sweep to the population it has authority over. A process reconciling against an external source has authority over rows originating from that source and none over anything else. Origin is not metadata; it is the boundary of the process's mandate.
And: never let a cleanup process double as a health check. If removing a sweep exposes stale rows from dead evaluators, the sweep was concealing an availability defect, not preventing one.
Chapter three
Load, suppression, and the arithmetic of an operator's burden
Suppression rate is the most quoted number in alarm management and one of the easiest to compute wrongly. A suppression rate measured over the wrong window is worse than no number at all, because it is believed.
The observed error
A view published a fleet suppression rate of 14.2%, with 2,992 alarms per hour reaching the board. Both figures were wrong, and wrong in the same way: the view counted suppression-tagged rows across a 24-hour window, but the trigger that writes the suppression tag had only existed for part of that window. Every hour before the trigger existed contributed rows to the denominator and none to the numerator.
Measured over a window lying entirely after tagging began:
| Measure | As published | Measured correctly |
|---|---|---|
| Suppression rate | 14.2% | 80.9% |
| Alarms per hour reaching the board | 2,992 | 1,131 |
The corrected figures are better news and a worse finding. Better, because suppression was working far more effectively than reported. Worse, because 1,131 per hour is still an unmanageable load, and the published number had been understating the effectiveness of the one control that was actually helping — which is precisely the kind of error that gets a working control switched off.
Calculation 2 · Suppression rate, computed honestly
-- The window must start at the LATER of (now - 24h) and the moment -- the tagging mechanism began writing. Publish the window with the rate. WITH w AS ( SELECT GREATEST(now() - interval '24 hours', (SELECT tagging_began_at FROM alarm_pipeline_meta)) AS win_start ) SELECT (SELECT win_start FROM w) AS window_start, EXTRACT(epoch FROM (now() - (SELECT win_start FROM w)))/3600 AS window_hours, COUNT(*) AS arrivals, COUNT(*) FILTER (WHERE archive_reason = 'state_event') AS suppressed, ROUND(100.0 * COUNT(*) FILTER (WHERE archive_reason = 'state_event') / NULLIF(COUNT(*),0), 1) AS suppressed_pct, ROUND(COUNT(*) FILTER (WHERE archive_reason IS NULL) / NULLIF(EXTRACT(epoch FROM (now()-(SELECT win_start FROM w)))/3600,0), 0) AS to_board_per_hour FROM alarms, w WHERE created_at >= w.win_start;
The rule embedded in that query is the one worth taking away: publish the window alongside the rate. A percentage with no stated measurement window is not a measurement, it is a rumour. The corrected view emits window_hours and a plain-language window_note on every row, so a reader can see that a rate computed over four hours is not comparable to one computed over twenty-four.
Load per operator — the number that actually matters
Fleet alarms per hour is a system property. Alarms per operator per hour is an ergonomic one, and it is the only version that predicts whether the board gets read.
Calculation 3 · Per-operator load and headroom
load_per_operator_hr = alarms_to_board_per_hour / operators_on_shift exceedance_factor = load_per_operator_hr / configured_threshold_per_hr -- Observed, after suppression was measured correctly: -- alarms_to_board_per_hour = 1,131 -- configured_threshold_per_hr = 60 -- -- 1 operator : 1,131 / 1 = 1,131/hr -> 18.9x threshold -- 3 operators : 1,131 / 3 = 377/hr -> 6.3x threshold
Deferred, and it matters. The 60-per-hour figure above is the threshold configured in the assessed platform. It is not a published standard value, and published alarm-management guidance sets materially lower long-term targets for sustained operator load. Before any exceedance factor from this chapter is quoted externally, the applicable benchmark must be read from the current published guidance and the arithmetic redone against it. The exceedance factors above are correct against the configured threshold and are presented on that basis only.
Two things follow regardless of which benchmark is eventually used. First, tripling the operators divides the load by three and does not change its order of magnitude — staffing is not the lever. Second, the entire value of the suppression control is visible in one subtraction: without it, load would have been roughly five times higher.
Where the load actually comes from
Board composition after remediation, from one assessed fleet:
| Priority | Rows | Share | Comment |
|---|---|---|---|
| Critical | 32 | 2.7% | The population the board exists for |
| High | 193 | 16.4% | Actionable, triaged |
| Medium | 837 | 71.2% | One transient work-state code is 51% of this bucket alone |
| Low | 35 | 3.0% | |
| Info | 78 | 6.6% | |
| Total | 1,175 | 100% |
Seventy-one percent of the board sat at Medium, and roughly half of that bucket was a single transient code with a 97.4% self-clear rate and a median duration of 0.56 minutes. By the test in Chapter 1, a condition that resolves itself in thirty-four seconds cannot be acted upon by a person and therefore is not an alarm — but it was correctly classified as a real condition rather than a state event, so it could not simply be swept.
That is a genuine design tension and it is worth naming rather than resolving glibly: a real condition that no human can act on in time still does not belong on an operator's board. The correct destination is a condition register with trending, not the alarm queue and not the bin.
The daylight problem — a worked example of a gate that must exist
An alarm catalogue promoted an inverter under-production code to High on the rationale "inverter producing nothing while the sun is up." The rationale is sound. The implementation had no way to know whether the sun was up.
Measured shortly after 21:00 local — full darkness — 582 of these were standing across two sites, with 637 fleet-wide. Promoting them to High would have put six hundred night-time alarms above the thirty-two Critical alarms the remediation had just made visible.
The promotion was held at Medium with the reason recorded as data in a catalogue-holds view, rather than either applied or silently dropped. That is the right handling of a decision that cannot yet be made correctly: the analysis is preserved, the blocker is named, and the unsafe change is not shipped.
Calculation 4 · The daylight gate
-- Promote a sun-dependent code only when irradiance supports the premise. -- 50 W/m2 is a deliberately conservative floor: comfortably above night, -- comfortably below any level at which real generation is expected. SELECT a.alarm_id, a.site_id, a.event_name, i.poa_wm2, CASE WHEN i.poa_wm2 IS NULL THEN 'hold - no irradiance reference' WHEN i.poa_wm2 < 50 THEN 'hold - dark' ELSE 'promote' END AS gate_decision FROM alarms a LEFT JOIN LATERAL ( SELECT poa_wm2 FROM site_irradiance_history WHERE site_id = a.site_id AND observed_at <= a.raised_at ORDER BY observed_at DESC LIMIT 1 ) i ON true WHERE a.event_name IN (SELECT event_name FROM alarm_catalog WHERE requires_daylight IS TRUE);
Note the three-way outcome. No irradiance reference is a hold, not a promote. A gate that defaults to "promote" when it cannot evaluate its own premise is not a gate. And a site with no irradiance reference is a gap on a list, not an exemption — the same principle as the missing nameplate row in EC-WP-1100.
Chapter four
Priority resolution, and who is allowed to decide
A priority classifier read a JSON blob posted by the browser, then a regex, then returned "Low." A 249-code reviewed catalogue existed and was never consulted. A governed override table with a named approver and a written reason was written by two functions and read by none.
The defect, stated as an authority problem
This is not primarily a code defect. It is a question about who decides the priority of an alarm, answered wrongly by the resolution order.
There were three sources of priority in the system, and they carry very different authority:
▸ An approved override — a deliberate decision by a named approver with a written reason, recorded. The highest authority available.
▸ The reviewed catalogue — 249 codes worked through by engineers, with rationale. Considered judgement, applied consistently.
▸ A blob posted by the client — whatever the browser happened to send.
The resolution order in force was, in effect, the reverse of that list.
The correction
-- Resolve in the order authority actually runs. -- First match wins; the client blob is the fallback, never the override. CREATE FUNCTION fn_alarm_priority_of(p_code text, p_event text, p_payload jsonb) RETURNS text LANGUAGE sql STABLE AS $$ SELECT COALESCE( (SELECT priority FROM alarm_priority_override WHERE event_name = p_event AND approved_at IS NOT NULL ORDER BY approved_at DESC LIMIT 1), -- 1. approved override (SELECT priority FROM alarm_catalog WHERE event_name = p_event), -- 2. catalogue by event (SELECT priority FROM alarm_catalog WHERE code = p_code), -- 3. catalogue by code alarm_classify_row(p_payload) -- 4. legacy fallback ); $$;
What surfaced when authority was restored
Critical alarms moved from 17 to 32. The fifteen that appeared had been sitting at lower priorities:
| New Criticals | What they were | Catalogue evidence |
|---|---|---|
| 10 | Trip-coil supervision failure across three sites | Self-clear 0.0%, median duration 26 days |
| 4 | Inverters stopped by emergency stop at one site | — |
| 1 | Inverter major fault, unit offline | — |
Trip-coil supervision is the circuit that confirms a breaker can still be tripped. Ten of those had been standing at High, beneath a Medium bucket of transient chatter, with a measured median duration of twenty-six days and a self-clear rate of zero. That combination has exactly one interpretation.
Two catalogue decisions deliberately held
A key too coarse to act on
One catalogue entry was keyed on an event name that covered three materially different conditions — a feeder breaker open, a generic breaker open, and a high-voltage breaker open with the main power transformer isolated. Applying the catalogue's suggested priority would have moved the third off the Critical row along with the first two.
Held, with the reason recorded as data. The correct fix is a finer key, not a coarser decision, and shipping the coarse decision while waiting for the finer key would have removed a transformer isolation from the operator's Critical view.
The daylight promotion
Covered in Chapter 3. Held at Medium pending the gate.
The design rule. Every derived attribute needs a documented resolution order, and the order must run from highest authority to lowest with the first match winning.
Then two constraints that keep it honest. A client-supplied value can never outrank a reviewed one — anything the browser posts is an input, not a decision. And an override function must refuse a scope its own classifier cannot read: the corrected implementation rejects a site-scoped override outright rather than storing a record that will never be consulted, because a governed decision that silently does nothing is worse than a refused one.
Finally: a held decision is recorded with its reason, in a view anyone can query. The alternative is that the analysis is lost and the same question is reopened from scratch in six months.
Chapter five
States are not conditions
Eighty-nine percent of one operator board consisted of objects that asserted nothing wrong. Removing them removed no information and no alarm.
The single largest available improvement to most alarm boards is not tuning, not rationalisation, and not staffing. It is the removal of objects that were never alarms.
| Object | Asserts | Belongs |
|---|---|---|
| State event | A transition occurred. Nothing is wrong. Sunrise. Mode changed. Breaker closed as commanded. | Event log, sequence-of-events |
| Condition | A state persists and will clear on its own. Real, trendable, rarely actionable inside its own lifetime. | Condition register with trending |
| Alarm | A condition requiring a person to act, now. | The operator board |
On the assessed board, 6,965 of 7,839 rows were state events. One — a daily sunrise transition — had latched 2,092 times. The remediation swept them and installed a BEFORE INSERT trigger so they land as archived with an explicit reason rather than latching, which is the important half: a sweep without a gate is a chore that runs forever.
Calculation 5 · Classify the board before touching it
-- Run this BEFORE any suppression work. It sizes the prize -- and, more usefully, it tells you what is NOT the prize. SELECT COALESCE(c.class, 'UNCATALOGUED') AS class, COUNT(*) AS rows_on_board, ROUND(100.0*COUNT(*)/SUM(COUNT(*)) OVER (), 1) AS pct, COUNT(DISTINCT a.event_name) AS distinct_codes FROM alarms a LEFT JOIN alarm_catalog c ON c.event_name = a.event_name WHERE a.state IN ('active_unacked','acked') GROUP BY 1 ORDER BY rows_on_board DESC;
The UNCATALOGUED row is the one to read first. Codes on the board that no catalogue describes are, by definition, codes nobody has decided about. On one assessed board every uncatalogued row turned out to be the platform's own derived observations — objects that had never been through the review that governed everything else, because they had not existed when the review was done.
An honest note on suppression. "Suppression" is a word that makes safety-minded people uncomfortable, and it should. What is described here is not suppression in the sense of hiding a real condition. It is filing: putting an object where it belongs, with a recorded reason, where it remains queryable. Every swept row in the assessed system retained its full history and an explicit archive reason. Nothing was deleted. If your suppression mechanism deletes, it is not suppression, and Chapter 2 is about what that costs.
Chapter six
Key stability, latching, and why acknowledgement can stop meaning anything
An alarm key is an identity claim: this is the same condition you saw before. When the key changes and the condition does not, acknowledgement silently stops working.
Unstable keys
Three alarm generators embedded epoch milliseconds in the key, so every evaluation cycle minted a new row for an unchanged condition. The consequences compound in a specific order, and only the last one is visible:
1. Row count grows without bound while the condition stands.
2. An operator acknowledges a row. The next evaluation creates a fresh unacknowledged row for the same condition.
3. The horn sounds again. The operator acknowledges again.
4. Within a shift, operators learn that acknowledgement does not work, and stop trusting the acknowledgement mechanism generally — including on alarms where it works fine.
Step four is the expensive one, and it is a behavioural failure produced by a data-model defect. Trust in a control is not per-alarm; it is global. A mechanism that fails visibly on some alarms is treated as unreliable on all of them.
Calculation 6 · Key stability test
-- A stable key produces ONE row per standing condition. -- rows_per_condition well above 1 = the key is minting duplicates. SELECT split_part(alarm_key, ':', 1) AS generator, site_id, COUNT(*) AS rows_open, COUNT(DISTINCT alarm_key) AS distinct_keys, ROUND(COUNT(*)::numeric / NULLIF(COUNT(DISTINCT split_part(alarm_key,':',2)),0), 1) AS rows_per_condition FROM alarms WHERE state IN ('active_unacked','acked') GROUP BY 1,2 HAVING COUNT(*) > COUNT(DISTINCT split_part(alarm_key,':',2)) ORDER BY rows_per_condition DESC;
The interim remediation collapses a repeat arrival for the same <generator>:<site> onto the open row. That is a compensating control and it was documented as one — remove it the day those generators emit a stable key. A compensating control with no removal condition attached becomes permanent architecture by default.
The opposite failure — alarms that stop re-latching
The mirror image is more dangerous, because it removes alarms rather than duplicating them.
Bulk archive operations — a baseline reset, a maintenance clear-down — archived thousands of rows wholesale. Alarms that had never actually cleared retained their alarm key. On the next ingest cycle the reconciler saw a key it already knew about, and the condition therefore never re-latched.
The consequence, measured on one assessed board: a turnover screen reading "0 critical, oldest critical: none" while the underlying plant carried multiple standing critical conditions, including breaker trips and, at one site, an isolated main power transformer.
The rule this produces. A bulk archive must either verify that each condition has genuinely cleared at source, or invalidate the keys it archives so the condition can re-latch on the next cycle. Silently retaining keys for conditions that were never resolved converts an administrative action into permanent blindness — and the blindness is invisible, because the screen reports zero.
Standing alarm age — the measurement nobody publishes
Of 2,126 alarms on one assessed board, 2,096 were genuinely still asserting. But 257 had been asserting for more than a week, and the tail is where the findings live:
| Days asserting | Priority | What it was |
|---|---|---|
| 458 | Critical | A breaker status point |
| 307 | High | Breaker trip-coil supervision |
| 307 | High | Breaker trip-coil supervision, second unit |
| 306 | High | Breaker spring-charge supervision |
| 68 | Critical | Four feeder breakers, trip coil and relay trip |
A breaker status Critical asserting for fifteen months is either a defect nobody closed or an instrument that has been lying for fifteen months. Both are findings, and the board cannot tell you which.
Age must be computed from the source clock — the timestamp the upstream system assigned when the condition began — not from the local row's creation time, which resets whenever the local store is rebuilt. Where the source clock is unavailable and a local fallback is used, the row is labelled as using a fallback, because an age computed from the wrong clock is a number that will be quoted.
Chapter seven
The monitor that stops
A stopped evaluator produces exactly the same board as a plant with nothing wrong. Every alarm system in this industry rests on the assumption that silence means health, and almost none of them test it.
This is the alarm-domain instance of the general failure described in EC-WP-1100: a system's inability to distinguish nothing is wrong from nothing is arriving. In the alarm domain it has a sharper edge, because the alarm board is the control that is supposed to catch everything else failing.
Observed case
A storage state-of-charge monitor stopped at 07:50. At 11:29 — nearly four hours later — the board still displayed a site at "6.7%, at or below 10%." The plant actually read 19.86% and had recovered hours earlier.
The alarm was not wrong when it was raised. It became wrong, silently, and there was no mechanism by which it could stop being displayed as current.
The three-layer answer
| Layer | What it does | Failure it catches |
|---|---|---|
| Evaluator heartbeat | Each monitor writes a timestamp on every completed pass, whether or not it raised anything. A monitor that has not written in more than its own interval is down. | The evaluator stopped |
| Scheduler health | A view over the job scheduler reporting last-success and last-failure per job. | The evaluator was never invoked |
| Per-alarm freshness | Every standing alarm carries the time its condition was last independently confirmed. Past a threshold it renders as unconfirmed. | This specific claim is no longer supported |
The third layer is the one that is almost always missing, and it carries a design decision worth stating explicitly: an unconfirmed alarm is labelled, not cleared.
Clearing it asserts that the condition ended, which is unsupported — the monitor stopped, so nothing knows what the condition is doing. Leaving it displayed as current asserts that the condition is confirmed, which is equally unsupported. The only defensible rendering is the third one: this alarm stands, and we have not been able to confirm it for 3h 39m.
Calculation 7 · Monitor health and alarm freshness
-- Layer 1: which evaluators have missed their own interval? SELECT monitor_name, expected_interval_minutes, last_completed_at, ROUND(EXTRACT(epoch FROM (now()-last_completed_at))/60, 1) AS minutes_since, CASE WHEN now() - last_completed_at > (expected_interval_minutes * interval '2 minutes') THEN 'DOWN' ELSE 'ok' END AS health FROM monitor_heartbeat ORDER BY minutes_since DESC; -- Layer 3: standing alarms whose claim is no longer confirmed. SELECT a.alarm_id, a.site_id, a.event_name, a.priority, a.last_confirmed_at, ROUND(EXTRACT(epoch FROM (now()-a.last_confirmed_at))/60, 1) AS unconfirmed_min, CASE WHEN now() - a.last_confirmed_at > interval '30 minutes' THEN 'UNCONFIRMED - do not treat as current' ELSE 'confirmed' END AS freshness FROM alarms a WHERE a.state IN ('active_unacked','acked') ORDER BY unconfirmed_min DESC;
The guard ledger — and a caution about documentation
One assessed platform's operator documentation described a control in precise terms: "a database trigger refuses any low-state-of-charge row that carries no measurement or a dead-zero publish, and audits every refusal — see the guard ledger."
There was no trigger, no monitor, and no ledger table. The ledger panel rendered a raw database error where the audit trail should have been.
The control was subsequently built for real — refusal trigger, audit ledger, monitor with the documented thresholds, and a state view. On the first pass it correctly refused and recorded a genuine dead-zero publish at one site: state-of-charge reported as 0.00% with state-of-health and available-charge both absent. That is exactly the case the control was described as handling, and it had been passing through unrefused for as long as the documentation had been claiming otherwise.
The caution. Documentation that describes a control is not evidence that the control exists, and it is worse than silence — it stops anyone looking. Any control that appears in operator-facing documentation should have a query that demonstrates it, and that query should be run before the documentation ships. This one is ours.
Chapter eight
Ten questions for your own alarm system
Every one of these surfaced a real defect on the system described in this document. None requires a vendor.
| Question | What a bad answer means | |
|---|---|---|
| 1 | What percentage of your board is state events rather than conditions? | Above ~20% and the board is not being read. On one assessed system it was 89%. |
| 2 | Does any alarm your platform computes itself survive an ingest cycle? | If a reconciliation sweep runs against an upstream list, verify it directly. Do not assume. |
| 3 | What is your suppression rate, and over what window was it measured? | A rate without a stated window is not a measurement. Check when the tagging mechanism began. |
| 4 | What decides an alarm's priority, and in what order? | If a client-supplied value can outrank a reviewed catalogue, the catalogue is decorative. |
| 5 | How long has your oldest standing Critical been asserting? | Measured in months means either an unclosed defect or a lying instrument. |
| 6 | Does acknowledgement survive the next evaluation cycle? | If keys embed a timestamp, it does not — and operators already know. |
| 7 | After a bulk archive, can an uncleared condition re-latch? | If keys are retained, no. The board will report zero and be wrong. |
| 8 | If an evaluator stopped, how would the board look different? | If the answer is "it wouldn't," you have no monitor health. |
| 9 | Does any promotion rule depend on a premise the system cannot evaluate? | A daylight-dependent rule with no irradiance reference promotes a night-time flood. |
| 10 | Is every control described in your operator documentation demonstrable by query? | Documented-but-absent is worse than absent. It stops people looking. |
What good looks like. A board where every row is a condition a person can act on; a stated suppression rate with its window; a documented priority resolution order; standing-alarm age published and bounded; monitor health visible; and every described control demonstrable. None of that requires new software. Most of it requires deciding that the board will tell the truth even when the truth is unflattering.
Chapter nine
What RenewOps does about this
The reconciliation layer.Reads your monitoring stack. Checks it against ratings, agreements and the documents of record. Declines to publish what it cannot defend.
Seven failure modes, and the RenewOps alarm layer built to answer each. Everything below was measured on a live production instance between 24 and 26 August 2026 and is dated where the figure matters.
One definition of "on the board"
The board carries two states and no others: unacknowledged-active and acknowledged. Acknowledging an alarm no longer archives it. An alarm that has been acknowledged and is still asserting carries an explicit ACK'D · STILL ASSERTING badge, because "I have seen this" and "this is over" are two different statements and the board had been recording them as one.
The audible feed is scoped to unacknowledged-active only. An acknowledged alarm goes silent and stays visible — which is the whole point of an acknowledgement, and is not what a horn wired to the same population as the board would do.
Status is not an alarm, and cannot latch again
The board went from 7,839 rows to 1,175. Of the 6,965 status and informational rows removed, 48 were held deliberately. One daylight status event had been appearing on an operator's board 2,092 times.
The sweep is the smaller half of that fix. The larger half is a rule installed at ingest so those rows never latch again — a new arrival of that class lands archived with an explicit archive reason attached, rather than landing on the board and waiting for the next cleanup. A board that has to be swept is a board that will need sweeping again.
Criticals went from seventeen present and none rendering, to thirty-two rendering.
Priority resolves in the order authority actually runs
Effective priority resolves through three tiers, in the order the authority genuinely runs: an approved override first; then the reviewed cause catalogue, matched by event name and then by code; and only then a classifier fallback. A priority posted by the browser can no longer contradict a catalogued code, which it previously could and did.
The write it refuses. A site-scoped priority override the classifier cannot read is rejected at the point of writing. The alternative — accepting it, storing it, and letting it silently do nothing — produces an override register that looks populated and governs nothing. An override that cannot take effect is worse than an override that was never made, because somebody will later cite it.
A vocabulary check reports any priority word outside the known set. Without it, a new value arriving from upstream sorts quietly to the bottom of every board and nobody is told.
Held, with the reason held as data
Two catalogue promotions are deliberately not applied, and the reason for each sits in a holds view rather than in a ticket:
▸ One catalogue key covers three different breaker conditions, including an isolated main transformer. Promoting the key would have moved that condition off Critical.
▸ 582 instances of a second code were standing after dark. Promoting a night-time flood to High would have buried the thirty-two Criticals underneath it.
The daylight gate that would resolve the second case is designed and not built. It is carried as an open item, not as a completed one. A hold with its reason recorded is a decision; a hold with no reason recorded is an omission that will be reversed by whoever inherits it.
Rollup that discards nothing
A grouped alarm takes the worst priority among its children plus the full child count. Nothing is discarded. The parent row also carries a priority-mix qualifier on its face: an operator reads "High (High, Medium)" and knows immediately that the group is not uniform. Five parent rows currently merge disagreeing priorities, and each states so, with asset counts and standing hours beside it.
A fix that was withdrawn before it ran. A proposed deduplication would have kept three assets at High or eleven at Medium and discarded the other. It was withdrawn. Under-reporting both count and severity is worse than staleness, because staleness at least announces itself in the timestamp. Nothing about the discarded rows would have.
Suppression measured against its own window
Suppression rate is published with the window it was measured over — from the later of twenty-four hours ago and the moment suppression tagging began, with the window length printed beside the figure. That correction moved a reported 14.2 per cent to a measured 80.9 per cent. The number had not changed. The denominator had.
At 80.9 per cent suppressed, 1,131 alarms per hour were still reaching the board.
Deferred. Alarm load is computed against a threshold of sixty alarms per operator-hour, which is a value configured in this platform. Whether that figure matches any published alarm-management guidance has not been verified against the source document, and it is not stated here as a standard.
The monitor that stops
The system publishes its own heartbeat, its monitor health, and the health of its scheduled jobs. Per-alarm freshness labels an unconfirmed alarm rather than clearing it. Two clocks are named separately: minutes since the newest onset, and minutes since the newest confirmation. A stopped feed cannot hide behind a calm board when the board publishes how long it has been since anything confirmed anything.
This matters because of what was found underneath it. A close-out sweep had been destroying every alarm RenewOps raised itself — 548 voltage-band and 246 schedule-deviation alarms raised and deleted, each stamped as though the plant had returned to normal, not one of them ever seen by an operator. The sweep now touches only alarms that originated upstream. On 25 August 2026, for the first time in the platform's life, sixteen voltage alarms stood across nine sites. Twelve were acknowledged within the hour.
The alarm system was not failing to detect. It was detecting correctly and then deleting the evidence on a schedule, and every screen downstream reported a quiet fleet.
Acknowledge what you say you are acknowledging
A bucket acknowledgement acknowledges the whole bucket server-side and returns the count actually changed. It refuses an unattributed call, refuses an unrecognised priority, never writes an archive stamp, and writes its own bulk-acknowledgement record in the same transaction. The confirmation dialogue shows the true bucket size. Previously, "Ack all (1,707)" acknowledged the hundred rows the browser had loaded and reported success.
What remains open
| Item | Position as of 26 August 2026 |
|---|---|
| Cause catalogue disagreement | 131 of 249 catalogued codes disagree with the priority currently in use, including a breaker trip-coil monitor sitting at High. Documented, not yet reconciled. |
| Override store empty | The governed override path exists and refuses bad writes. Nothing has used it. Applying an override today writes an audit trail and changes nothing until the classifier reads it. |
| Platform-generated flood | Forecast-deviation alarms embed a percentage in the description, so every distinct percentage is a distinct code and none can match the catalogue. A nightly closure job stops them accumulating; the key shape is still wrong. |
| Attribution | Content is guarded well. Identity is a string the caller supplies. Who acknowledged, overrode, or held is not independently verified. |
Figures measured 24–26 August 2026 on a live production instance and dated in the text. They describe that instance on those dates and are not a claim about any other fleet.
The firm
About Energy Compliance, Inc.
Energy Compliance, Inc. is an independent regulatory compliance and advisory firm serving the energy sector, with a focus on NERC, FERC, and RTO/ISO compliance and a particular concentration in the ERCOT and Texas PUCT markets. The firm helps registered entities and prospective registrants navigate the full compliance lifecycle — registration, program design, evidence development, RSAW production, mitigation, and audit defense.
The approach is research-analyst-first. Every conclusion is tied to an authoritative source, every narrative is evidence-backed, and every deliverable is built to survive regulator scrutiny. Engagements are led by a single senior practitioner with regulator-side experience. We do not staff for billable hours. We staff for outcomes. Where automation can replace manual work, we build the automation. Where senior judgment is required, the senior is in the room.
That discipline extends to a portfolio of compliance technology, of which RenewOps — the reconciliation layer described throughout this series — is one part.
Rob Smith — Founder & Principal
More than thirty years on every side of the North American Bulk Electric System. Control-room operations as Reliability Coordinator, Transmission Operator and Power System Operator. Senior compliance auditor and subject matter expert for NERC Reliability Standards — auditing grid facilities, evaluating mitigation adequacy, and supporting the development of violation notifications and settlements as part of FERC-directed enforcement actions, from inside the regulator's process. Overseas, a regulatory audit in the Sultanate of Oman conducted against the Sultanate's Sector Law and Grid Code.
MSL, Corporate Compliance — Fordham Law · MS, Energy Management · BAS, Mechanical Engineering (Metallurgy minor), University of Florida · BAS, Energy Management · AAS, Power Plant Technology and Electrical Transmission System Technology, Bismarck State College
Auditing teaches one habit that never leaves: before you believe a number, ask what it would look like if it were wrong.
Where to start
An assessment points our defect battery at your live monitoring stack and hands you the findings — each with a source, a timestamp, and the question that resolves it — whether or not you buy anything afterward. Nobody buys a reconciliation layer before they know they need one, and the only honest way to find out is to look.
Fleet Exposure Read
Your availability constructs reconciled against your executed agreements, returning a per-site computability position — ready, partial, blocked, or undefined — with the specific blocker named for each site that cannot compute.
Monitoring Truth Assessment
The full defect battery against your live monitoring stack. Every finding returns with a source, a timestamp, and the question that resolves it. Where we cannot verify something, it comes back marked deferred with the specific question, not softened into a maybe.
Evidence Provenance Review
Your compliance-relevant signals traced to source and sorted into citable and not, with the seeded, the derived, and the self-referential separated out from the issued.
RenewOps
The reconciliation layer itself, deployed against your fleet: contract terms parsed to structure, signals resolved to roles, every tile carrying its own denominator, its own provenance, and its own age — and refusing to compute, visibly, when it cannot.
Energy Compliance, Inc.
[email protected] · energycomplianceinc.com · 763.438.4427Bloomington, Minnesota
To discuss how any of this applies to a specific fleet, or to request other references from the Energy Compliance library, visit energycomplianceinc.com.
Rigorous Compliance.Defensible Programs.
energycomplianceinc.com