A Rules Engine That Measures Its Own Usefulness

Chapter one

What a rule has to earn

A rule occupies an operator's attention, which is the scarcest resource in a control room. It has to pay rent, and the rent has to be measured.

The four properties

Most rule engines test the first property and assume the other three. The assessment described here failed on all three of the assumed ones while passing convincingly on the one that was measured.

The rule record

A rule in NORA is a row, and the fields it carries are the argument of this chapter:

-- What a rule has to declare about itself. signal -- constrained vocabulary; see Chapter 7 threshold -- refused below a floor; see below evaluation_interval -- refused under five minutes quiet_period -- and the KEY it is scoped to; see Chapter 5 prompt_text -- what the operator is actually asked enabled -- and enabled is not the same as capable not_needed_rate -- published, per rule, visible to everyone

Two of those carry refusals rather than defaults, and both refusals exist because of observed failures. An evaluation interval under five minutes is refused — a rule evaluating faster than an operator can respond is a flood generator with a threshold. And a rule change with no reason is refused, because a threshold that moved with no recorded rationale cannot be reviewed later, and thresholds drift toward whatever silences the complaints.

The distinction the rest of this paper turns on. enabled means somebody switched the rule on. It does not mean the rule can fire. Four of the seven rules assessed were enabled, healthy-looking, and structurally incapable of producing a single prompt. Nothing in the interface distinguished them from rules that simply had nothing to report.

Chapter two

The not-needed rate

The measurement that lets an alarm set shrink. It is not a technical innovation; it is a willingness to record an answer nobody wants to hear.

The mechanic

When a rule surfaces something, the operator is asked to close it with one of a small set of outcomes. One of those outcomes is this did not need surfacing. The proportion of closures carrying that answer is the rule's not-needed rate, and it is published in the rule list beside the rule's name.

Calculation 1 · Not-needed rate, with a confidence gate

SELECT r.rule_key, r.enabled, COUNT(a.*) AS answers, COUNT(*) FILTER (WHERE a.outcome = 'not_needed') AS not_needed, ROUND(100.0 * COUNT(*) FILTER (WHERE a.outcome = 'not_needed') / NULLIF(COUNT(a.*),0), 1) AS not_needed_pct, -- do not retire a rule on three answers (COUNT(a.*) >= 20) AS sample_sufficient, CASE WHEN COUNT(a.*) < 20 THEN 'observe' WHEN COUNT(*) FILTER (WHERE a.outcome='not_needed') * 100.0 / COUNT(a.*) >= 40 THEN 'RETIRE' WHEN COUNT(*) FILTER (WHERE a.outcome='not_needed') * 100.0 / COUNT(a.*) >= 15 THEN 'retune' ELSE 'earning its place' END AS verdict FROM nora_rule r LEFT JOIN nora_answer a ON a.rule_key = r.rule_key GROUP BY r.rule_key, r.enabled ORDER BY not_needed_pct DESC NULLS LAST;

Reading the verdict

The observed rules all returned 0–1.5%, well inside "earning its place" — but three of the six had fewer than twenty answers, which is exactly the case the sample gate exists for. A rate computed from six answers is not evidence that a rule is good; it is evidence that a rule has fired six times.

The trap this measurement sets for itself. A rule that cannot fire has no answers, so it has no not-needed rate, so it never appears in the retirement queue. Its silence reads as virtue. Four of the seven rules assessed were sitting in exactly that position — and the not-needed rate, on its own, would have protected them indefinitely. That is Chapter 3.

Chapter three

Four rules that could never fire

The candidate view assembled signals from four producers and inner-joined them to the rule table. Any rule whose signal had no producer was silently dropped from the result — enabled, visible, and structurally incapable of firing.

The mechanism

-- Simplified. The join is the defect. SELECT r.rule_key, s.site_id, s.value FROM ( SELECT 'comms_condition' AS signal, ... FROM ... UNION SELECT 'alarm_flood' AS signal, ... FROM ... UNION SELECT 'protection_long' AS signal, ... FROM ... UNION SELECT 'compliance_due' AS signal, ... FROM ... ) s INNER JOIN nora_prompt_rule r ON r.signal = s.signal; -- <-- here -- Rules whose signal is not in the union simply vanish. -- No error. No warning. No row in any diagnostic.

Four signals had rules but no producer: alarm rate, stale data, availability below guarantee, and irradiance reference. Each rule existed, was enabled, and had a threshold. The inner join removed them from consideration on every evaluation cycle.

What that concealed

Because the rules could not fire, the health page reported "quiet — no independent check available." At the same moment, measured directly:

The last row deserves its own note. A threshold of zero cannot be exceeded. Even had the producer existed and the join succeeded, that rule would still never have fired — a second, independent reason for the same silence, hidden behind the first.

Calculation 2 · Producer coverage — run this on any rule engine

-- Every rule whose signal no producer emits. Should return zero rows. SELECT r.rule_key, r.signal, r.enabled, r.threshold, 'NO PRODUCER' AS defect FROM nora_prompt_rule r WHERE NOT EXISTS (SELECT 1 FROM nora_signal_producer p WHERE p.signal = r.signal) UNION ALL -- Every rule that CAN fire but has not, for longer than its own interval. SELECT r.rule_key, r.signal, r.enabled, r.threshold, 'ENABLED BUT SILENT' AS defect FROM nora_prompt_rule r WHERE r.enabled AND NOT EXISTS (SELECT 1 FROM nora_prompt pr WHERE pr.rule_key = r.rule_key AND pr.created_at > now() - interval '30 days');

The second half of that query is the more general control. An enabled rule that has produced nothing in thirty days is making a claim — that its condition has not occurred — and that claim deserves a check rather than an assumption.

The design rule

A rule declares its producer, and the engine refuses a rule whose producer does not exist.

Not a warning, not a log line. A refusal at write time, in the same function that already refuses an interval under five minutes. A rule that cannot fire should be impossible to save, because once saved it is indistinguishable from a rule with nothing to report.

And the health surface must separate three states: evaluated, no condition found · not evaluated, producer missing · not evaluated, evaluator down. Collapsing those into "quiet" is the same substitution failure that runs through this entire body of work — an absence rendered as a reassuring value.

Chapter four

The rule that fired on the healthy sites

A communications-loss rule counted communications-loss alarms. A site that has lost communications cannot send one.

The inversion

The rule's evidence was the arrival of alarms of a particular type from a site. Its premise — more comms alarms means worse communications — is reasonable and, over the middle of the range, true.

At the boundary it inverts completely. A site whose link has failed entirely sends nothing, including comms alarms. Its count is zero. Zero is the healthiest possible value under the rule's logic.

Observed behaviour: the rule fired on four sites whose links were reported healthy, and stayed silent on the four sites that were completely dark.

The rule was not merely wrong. It was precisely, systematically anti-correlated with the condition it existed to detect.

The general form

This is worth generalising, because it is a whole class rather than one bug:

The inversion test. For any rule, ask: if the condition were completely true, would the evidence still arrive?

If the evidence is generated by the thing that is failing, the answer is no, and the rule inverts at exactly the point where it matters most. Rules built on counts of incoming events are the usual carriers. Rules built on absence of expected events are usually safe.

The repair was to repoint the rule at a link-health view that separates rows arriving from values arriving — a distinction that survives total failure because it is built on expected arrivals rather than observed ones.

Why it survived review

Because it fired. A rule that produces prompts looks alive, and the prompts it produced were not obviously wrong — the sites it named did have communications events. Nobody compares the set of sites a rule fires on against the set of sites that are actually failing, because doing so requires an independent source of truth about failure, which is the thing the rule was supposed to be.

Chapter five

Quiet periods, and what an operator actually dismissed

One operator answered eleven prompts in ninety-nine seconds. The quiet period that followed muted all twenty-eight protection conditions across the fleet for a week.

What happened

Eleven prompts in ninety-nine seconds is nine seconds per prompt. That is not consideration; it is clearing a queue. The interface asked eleven similar questions in succession and the operator did the rational thing.

The quiet period then did what it was configured to do: suppress further prompts. But it was keyed per site rather than per condition, so dismissing eleven prompts about specific protection elements silenced every protection condition at those sites, including conditions the operator had never been shown.

The two design errors, separately

Scope. A dismissal is a statement about the thing dismissed. It says nothing about a different condition on the same asset, and considerably less than nothing about a different asset. The quiet period must be keyed to the narrowest identifier that describes what the operator actually saw — condition and asset, not category and site.

Duration versus deliberation. Seven days is a reasonable quiet period for a considered dismissal and an unreasonable one for a nine-second one. The system has the information required to tell them apart — it knows how long the prompt was on screen — and did not use it.

Calculation 3 · Quiet period scaled to deliberation

-- Detect queue-clearing, and let the quiet period reflect it. WITH answered AS ( SELECT actor_code, rule_key, prompt_id, answered_at, EXTRACT(epoch FROM (answered_at - shown_at)) AS seconds_on_screen, COUNT(*) OVER (PARTITION BY actor_code ORDER BY answered_at RANGE BETWEEN interval '5 minutes' PRECEDING AND CURRENT ROW) AS answers_in_5min FROM nora_prompt_answer ) SELECT actor_code, prompt_id, seconds_on_screen, answers_in_5min, CASE WHEN seconds_on_screen < 15 OR answers_in_5min > 5 THEN interval '4 hours' -- queue-clearing: short mute ELSE interval '7 days' -- considered: full quiet period END AS quiet_period, (seconds_on_screen < 15 OR answers_in_5min > 5) AS rapid_dismissal FROM answered;

The rapid_dismissal flag is worth publishing independently of what it does to the quiet period. A rule accumulating rapid dismissals is telling you something the not-needed rate will not: operators are not evaluating it, which means its not-needed rate is measuring the interface rather than the rule.

The design rule. A dismissal binds only what was dismissed, and for a period proportionate to the attention it received.

Key the quiet period to condition-plus-asset. Scale the duration by observed deliberation. Publish rapid-dismissal rate per rule alongside not-needed rate — and read the two together, because a rule with a low not-needed rate and a high rapid-dismissal rate is not earning its place, it is being waved through.

The prompt itself is part of the rule

Eleven near-identical prompts in a row is an interface failure, not an operator failure. Where a condition affects many assets simultaneously, the correct object is one prompt describing the group, with the members enumerated inside it — one decision, correctly scoped, recorded once against everything it covers.

Chapter six

The procedure library — retrieval that cites

When a rule surfaces something, the response draws on the entity's own controlled procedures, cited to document and section. The operator reads their procedure, not a vendor's idea of one.

What was loaded

The classification decision, and why it mattered

None of the ingested documents is a reliability standard. They are the entity's own procedures, which implement standards. That distinction determines what the retrieval layer is allowed to answer with.

An earlier load let automatic detection read internal document numbers as standard numbers, and filed eight procedures as standards. The consequence, had it stood: asked what a particular standard requires, the assistant would have answered with the entity's own procedure rather than the standard text already in the index.

That is a provenance failure in the sense of EC-WP-1100 — a document of one epistemic status rendered as another — and in a compliance context it is the expensive kind. A procedure describes what an entity chose to do. A standard describes what is required. Confusing them in an evidence conversation is how an entity ends up citing itself.

Two live findings the load surfaced in the documents themselves.

Ninety-nine of the 121 still carried a template document number rather than a real one. An unnumbered controlled document is an audit finding on its own, and it was raised ahead of the library's effective date rather than discovered later.

The library was future-dated. Every document carried an effective date ahead of the load. The library existed and was not yet in force — a distinction the retrieval layer must carry, because answering an operator today from a procedure that takes effect next month is wrong in a way that looks entirely right.

The ingest path, and why it is audited

Documents enter through a function that re-verifies the operator's role against the roster on every call and writes an audit row per ingest. That is deliberate: a procedure library is an evidence source, and an evidence source with an unaudited write path is not one.

Supersession is explicit — a replacing document names the document it retires, and the retired version stays queryable with a retired status rather than being deleted. A procedure library that cannot show what was in force on a date in the past cannot support an event reconstruction, which is most of what it exists for.

Three defects in our own tooling, and the shape they shared

All three are the same shape: a failure that looked like a result. An empty list read as "nothing to do." A timeout read as an answer. A dropped connection read as the end of the work.

That sentence is the thesis of this entire body of work, arrived at from a completely different direction. The fixes were correspondingly simple: retry and then abort rather than trusting an empty answer; one worker; catch transport failures and retry them, so that only a real answer from the function — good or bad — ends the loop.

Chapter seven

What the vocabulary permits

The signal vocabulary is a constraint with exactly nine permitted values. Widening it is a governance decision about what the system is allowed to assert, not a repair.

Why constrain the vocabulary at all

An unconstrained signal field means every rule can invent its own vocabulary, producers and consumers drift apart silently, and the inner-join failure of Chapter 3 becomes undiagnosable. The constraint is what makes producer coverage checkable.

It also has a second effect, less obvious and more important: it forces the question of what the system is entitled to claim. Each permitted signal is a category of assertion the platform is prepared to make to an operator.

The signal that is missing

None of the nine permitted signals describes: this value arrived on time, from a healthy link, and is not physically possible.

That gap is why the plausibility failures in EC-WP-1100 — a meter reading orders of magnitude above nameplate, mixed units in a single column, a setpoint of a different order from the quantity it regulates — could not raise an operator prompt. The rules engine had no vocabulary for the claim.

The proposed shape is straightforward: a plausibility signal, a per-site band table sourced from voltage class and nameplate, an audit row per refusal, and the same effectiveness measurement as the other rules.

It was proposed and deliberately not added. Widening the constraint governs what NORA is allowed to say to an operator. That is a platform decision with consequences beyond one rule, and it belongs to the person accountable for the platform rather than to whoever happened to be in the codebase that night.

Recording it as a named, pending decision — rather than shipping it because it was obviously right — is the same discipline as the held catalogue decisions in EC-WP-1101. The analysis is preserved, the blocker is named, and nobody has to rediscover the argument.

What the rules caught once they could fire

After the producers were built and the rules repointed:

The third row is the one to dwell on. A rule returning zero after its producer exists and its logic is verified is a meaningful zero. Before that, the same zero was meaningless. The output is identical; only the epistemic status changed.

Chapter eight

Ten questions for your own rules engine

Every alarm system grows monotonically until somebody measures whether the rules are worth having. The measurement is not hard. Publishing it is.

Chapter nine

What RenewOps does about this

The reconciliation layer.Reads your monitoring stack. Checks it against ratings, agreements and the documents of record. Declines to publish what it cannot defend.

NORA is the rules engine inside RenewOps, and the subject of this reference. What follows is what it does, what it refuses to do, and the four things it got wrong that its own instrumentation caught.

The measurement that earns a rule its place

Every rule is scored by its not-needed rate — the share of its prompts an operator answered "not needed." On 25 August 2026 six enabled rules all measured between zero and 1.5 per cent, with answer counts of 66, 21, 13, 12, 7 and 6. Each was reported as earning its place, and the number rather than an opinion is what earned it.

A rule has three legible states, not two. Enabled. Disabled, with the reason written onto the rule record. And NOT WIRED — a rule that exists but has no candidate branch and therefore could never fire under any input. That third state is the one this reference argues matters most, because a rule with no candidate branch and a rule that is quiet because the fleet is healthy look identical from outside and mean opposite things.

How that was found. On 24 August 2026 four of seven rules structurally could not fire, while the screen printed quiet — no independent check available for each of them. An operations manager reading that screen would reasonably have concluded the fleet was clean on four dimensions it was not being checked on at all. Enabled rules went from six to nine on 25 August, and the wiring state is now rendered separately from the firing state.

The inverted rule

The communications rule counted comms-loss alarms. It fired on healthy sites and stayed silent on dark ones, for the reason that reads as obvious once stated: a site that has gone dark cannot send you an alarm saying it has gone dark. The rule was measuring the presence of a message from a site, which is close to the opposite of the condition it was named for.

It was replaced by a stale-feed rule that asks the inverse question — how long since this feed last said anything — which covers dark sites by construction. On enable it flagged twelve of thirty feeds stale, from 796 days down to 3.9 days, per feed and per site.

Two gates every candidate branch has to pass

A rule comparing a partial day's measurement against a full day's model made the entire fleet look faulted every morning and recover every afternoon. Seventeen sites read suspect on one day and normal on the next, with nothing having changed at any plant.

Two requirements came out of that, and they now apply to any candidate branch:

1. A completed-day gate. A comparison against a daily model does not run until the day it describes has finished.

2. A persistence test. A condition that appears once and clears is not a finding. The genuinely faulted references are the persistent ones, and the rule is written to say so.

Re-enabled on completed days, the rule caught four genuine site-days in seven — sites generating between 1,000 and 1,800 MWh against zero reported sunlight. That is the finding the morning artifact had been burying.

The rules it declines to enable

The configuration it refuses. The rule-writing path rejects settings that would make a rule useless — an evaluation interval under five minutes is refused outright. The engine guards its own content. It does not guard the identity of who changed a rule, which is a caller-supplied string, and that is stated below rather than glossed.

The quiet period, and what it cost

RenewOps applies a per-site quiet period to suppress repeat prompts for the same condition. It worked, and then it over-worked: one user answering eleven prompts in ninety-nine seconds muted twenty-eight standing protection conditions at once. The quiet period has been shortened. Scoping it per condition rather than per site is owed and not done.

This is the failure mode a not-needed rate is blind to. A rule that never prompts scores perfectly on a measure of whether its prompts were useful.

What the instrumentation cannot tell you

The not-needed rate is a real measurement taken over an attribution trail that is not authenticated. The responder's name is client-supplied. The measurement is sound as a measure of how often prompts were dismissed; it is not evidence of who dismissed them.

And no rule has yet been retired on the measure. The lifecycle this reference can evidence runs enabled → disabled-with-reason → not-wired. Retirement is the step the not-needed rate exists to justify, and it has not been exercised, because on the fleet measured, every enabled rule was earning its place. A ratchet that has never had to release is not yet proven to release.

A rules engine that publishes how often it was ignored is making an argument against its own headcount. That is the point. The engines that grow without limit are the ones that never had to make it.

Figures measured 24–26 August 2026 on a live production instance and dated in the text. They describe that instance on those dates and are not a claim about any other fleet.

The firm

About Energy Compliance, Inc.

Energy Compliance, Inc. is an independent regulatory compliance and advisory firm serving the energy sector, with a focus on NERC, FERC, and RTO/ISO compliance and a particular concentration in the ERCOT and Texas PUCT markets. The firm helps registered entities and prospective registrants navigate the full compliance lifecycle — registration, program design, evidence development, RSAW production, mitigation, and audit defense.

The approach is research-analyst-first. Every conclusion is tied to an authoritative source, every narrative is evidence-backed, and every deliverable is built to survive regulator scrutiny. Engagements are led by a single senior practitioner with regulator-side experience. We do not staff for billable hours. We staff for outcomes. Where automation can replace manual work, we build the automation. Where senior judgment is required, the senior is in the room.

That discipline extends to a portfolio of compliance technology, of which RenewOps — the reconciliation layer described throughout this series — is one part.

Rob Smith — Founder & Principal

More than thirty years on every side of the North American Bulk Electric System. Control-room operations as Reliability Coordinator, Transmission Operator and Power System Operator. Senior compliance auditor and subject matter expert for NERC Reliability Standards — auditing grid facilities, evaluating mitigation adequacy, and supporting the development of violation notifications and settlements as part of FERC-directed enforcement actions, from inside the regulator's process. Overseas, a regulatory audit in the Sultanate of Oman conducted against the Sultanate's Sector Law and Grid Code.

MSL, Corporate Compliance — Fordham Law · MS, Energy Management · BAS, Mechanical Engineering (Metallurgy minor), University of Florida · BAS, Energy Management · AAS, Power Plant Technology and Electrical Transmission System Technology, Bismarck State College

Auditing teaches one habit that never leaves: before you believe a number, ask what it would look like if it were wrong.

Where to start

An assessment points our defect battery at your live monitoring stack and hands you the findings — each with a source, a timestamp, and the question that resolves it — whether or not you buy anything afterward. Nobody buys a reconciliation layer before they know they need one, and the only honest way to find out is to look.

Fleet Exposure Read

Your availability constructs reconciled against your executed agreements, returning a per-site computability position — ready, partial, blocked, or undefined — with the specific blocker named for each site that cannot compute.

Monitoring Truth Assessment

The full defect battery against your live monitoring stack. Every finding returns with a source, a timestamp, and the question that resolves it. Where we cannot verify something, it comes back marked deferred with the specific question, not softened into a maybe.

Evidence Provenance Review

Your compliance-relevant signals traced to source and sorted into citable and not, with the seeded, the derived, and the self-referential separated out from the issued.

RenewOps

The reconciliation layer itself, deployed against your fleet: contract terms parsed to structure, signals resolved to roles, every tile carrying its own denominator, its own provenance, and its own age — and refusing to compute, visibly, when it cannot.

Energy Compliance, Inc.

[email protected] · energycomplianceinc.com · 763.438.4427Bloomington, Minnesota

To discuss how any of this applies to a specific fleet, or to request other references from the Energy Compliance library, visit energycomplianceinc.com.

Rigorous Compliance.Defensible Programs.

energycomplianceinc.com