Engineering Studies: Pre- & Post-COD

Foreword

Foreword

This professional reference is one of a series Energy Compliance, Inc. publishes for registered entities and the people who run their compliance programs.

I’ve spent more than thirty years on every side of the bulk electric system. I’ve operated control centers as a Reliability Coordinator, Transmission Operator, and Power System Operator. I’ve audited grid facilities and signed off on findings as a senior compliance auditor. I’ve worked enforcement matters from inside the regulator’s process. For the last several years I’ve advised registered entities directly through the firm I founded.

The entities that do reliability well share a common habit. They take the standards seriously without confusing them with reliability itself. They know that a NERC Reliability Standard is a floor, not a ceiling. They know that compliance is something an auditor evaluates, but reliability is something a system either delivers or doesn’t. They prepare for audits by building programs that survive real questions, not binders that look thick.

That’s the perspective these references try to share. Each one focuses on a single topic. A standard family, an operational function, a regulatory framework, or an emerging industry challenge. Each one walks through how the topic actually works.

These references are written for the compliance manager who wants to understand the system, not just memorize requirements. For the legal counsel who has to brief a board honestly. For the senior operator who’s been told that compliance and reliability are the same thing and suspects they aren’t. And for the new compliance hire who got handed a binder and told good luck.

These references aren’t marketing material disguised as content. They’re the result of three decades of doing this work and watching it succeed and fail. I’ve written them in the same voice I use in a control room or in front of a Regional Entity audit team. Direct, evidence-grounded, honest about what the standards do and do not require.

Energy Compliance exists because most of the consulting offered to registered entities today is structured for billable hours rather than for outcomes. Every engagement is led by one senior practitioner. We don’t bring five people to a meeting that needs one. We automate the work that should be automated. We apply senior judgment to the work that requires it. If that approach matches what you’re looking for in a compliance partner, the back of this reference has our contact information.

If not, the reference still belongs to you. Take what’s useful. Apply it well. And remember the only test that ultimately matters: when the system needs to perform, does it?

Rob Smith, Founder, Energy Compliance, Inc.

EC-WP-504 Engineering Studies

Chapter 1

Engineering Studies Within the Reliability Framework Post-commercial operation studies address a different but not less important requirement. The

performance, operational patterns and real system conditions that arise during the operation of a generation resource after it has been commissioned can differ from those assumed during the planning

stage. The post-commercial studies enable reliability impact assessments to be re-visited, assumptions to be verified and any changes in the mutual impact of components on reliability to be discovered. They are therefore of great importance for maintaining situational awareness and for the ongoing verification of reliability analysis results with respect to the real conditions in the power system.

Reliability Engineering – Study and Oversight Part II provides readers with a broad overview of the place of reliability engineering studies within the framework of reliability oversight and offers possible solutions to meet the challenges arising from the increasing complexity of today’s systems. In addition, it describes the role of engineering studies in the context of reliability oversight activities. The oversight bodies make the necessary use of the results of these studies in order to understand the impact of the systems, to uncover potential sources of risk, and to ensure that the reliability criteria are consistently applied across different regions and technologies. Oversight is not concerned with the methodology of preparing studies, but rather with the validity of the scopes, hypotheses and timing that are chosen to assess the reliability risks.

This standard applies to individuals responsible for planning, interconnection, operation, compliance and/or oversight activities. It provides a structured foundation of knowledge of engineering studies as reliability artifacts that interfaces with NERC and RTO/ISO reliability requirements, and describes how such studies can be utilized to support the reliable interconnection and operation of new generation resources, without mandating specific engineering or operating activities.

The engineering studies required or expected for generation resources are one of the foundation blocks of the reliability framework for the Bulk Electric System. While these studies are technically complex to prepare, the purpose of these studies in the reliability context is not to identify the engineering design that must be built in order to meet the reliability criteria, but to determine whether the generation resource (new or existing) complies with the reliability assumptions and performance criteria built into the reliability framework. From the perspective of regulator, the engineering studies serve as a structured record of the reliability analysis performed at various timescales during both the planning and operational periods.

ERNH Engineering Studies Basis ERNH Reliability Planning Documents NERC Engineering Studies are closely related to planning and operating procedures dealing with power system adequacy, system operating capacity, and system response to credible contingencies. These procedures utilize analytical techniques for planning purposes in order to determine the expected performance of the power system to various future events. The studies and analyses supply the necessary technical information for the planning tools in order to convert reliability criteria into predicted system performance. In other words, the studies provide the analysis which will predict the system performance to meet the criteria, without specifying the design of the facilities.

Studies for Generation Resources in Interconnection/Transmission Regions A Regional Transmission Organization (RTO) or an Independent System Operator (ISO) typically defines the studies needed for each generation resource due to the interconnection and operating procedures in these transmission regions. Studies may differ from one RTO/ISO region to another, but are generally used to determine whether new and existing generation will impact reliability and the R/E Vulc Stability transfer capabilities so that their connections to the transmission system can be made in accordance with regional generation planning studies. From a reliability perspective, these regional study processes, in conjunction with NERC Reliability Standards, provide a mechanism for covering those aspects of generation reliability relating to local system conditions that are not addressed within the broader framework of the grid-wide reliability disciplines.

Studies are a part of the engineering process, but differ from actual design in that they are concerned with system interactions rather than specifications of individual pieces of equipment. A large number of studies deal with analysis of the behavior of a generating resource in the interconnected system as a whole, and thus reflect its possible impact on the system, its response to various system occurrences and the effect of the inclusion of the new resource on the reliability indices. Reliability concerns with these studies are concerned with whether these issues have been properly addressed, and not so much with the techniques used in the studies.

One of the key features of engineering analyses within the reliability context is the necessity of making assumptions. Analyses are based on hypothetical system states, assumed operating modes and resource performance. Assumptions are used to make the analyses analytically tractable. Reliability review therefore has to consider not only the analysis results but also the appropriateness, clarity and consistency of the assumptions made. The risk of reliability consequences due to non-actualisation of assumptions is also an important aspect to be considered.

New engineerying work is also needed to fully address the need for coordination across all functional and organizational boundaries, such as for shared generation owners, transmission planners, operators, and reliability organizations. The reason is that all these groups need to have a systems view of the behavior of the grid, in order to support their operational decisions, since the reliability consequences of any one resource in the grid can extend far beyond that resource itself.

From reliability point of view, engineering studies are dynamic and they do not reflect “static” situation. This means that they reflect the system situation, the engineering approaches and the reliability aspects that are relevant to a given time. Because the system evolution may lead to different situation for which earlier studies are no longer applicable and new studies have to be done or earlier studies need to be updated. Reliability coverage therefore should be also considering this dynamic aspect of the system management as a normal and therefore not a failure situation.

This chapter has demonstrated that engineering studies are reliability analysis tools, which are compliant with NERC, RTO and ISO regulations. The rules and standards for the Bulk Electric System and the relationship of generation resources to those standards are described, emphasizing the fact that engineering studies are used as tools to prove that certain actions or projects meet reliability criteria or standards, rather than as design tools or prescriptions for performing work.

End-of-Chapter Summary

Engineering studies are used as analytical tools in the reliability framework to understand how resources in the generation system interact with the Bulk Electric System (BES) under various planning and operational scenarios. The studies are mandated under NERC Reliability Standards and processes initiated by RTOs and ISOs. These studies are used for reliability analysis of the future Bulk Electric System and do not prescribe how modifications must be engineered and performed.

by examining the system interaction, assumptions and coordination involved in the engineering studies (with focus on the reliability impacts and the clear separation between reviewing and performing the work). The chapter sets the stage for the reliability verification role of pre- and post-commercial operation studies.

FROM THE FIELD

Pre-COD studies tell you what the system will do. Post-COD studies tell you what it actually does. The gap between them is where most reliability surprises live.

An engineering study is not a check-the-box exercise. It is a reliability artifact. The study's conclusions get cited, audited, and acted on for years.

A study without verifiable assumptions, traceable methodology, and documented conclusions is not a study. It is documentation. The audit knows the difference.

Chapter 2

Pre-Commercial Operation Studies and Reliability Validation

Engineering studies that are performed prior to the commercial operation of a new resource are key to understanding how the new resource will interact with the Bulk Electric System. Reliability studies are a crucial tool in the planning and interconnection processes to determine whether a resource’s parameters are consistent with the grid reliability model. The purpose of the reliability studies is not to verify the technical design of the resource; rather, it is to ensure that reliability considerations are addressed before the resource begins operation. Pre-commercial operation studies are studies that are carried out during the planning and interconnection stages of the project, when the future generation sources are only planned and not yet in operation. Under these circumstances, they aim to assess the expected behaviour of the power system under various conditions that are dynamically simulated and which can include: forecasted load and generation, network state and operating conditions, various scenarios of system contingencies etc. From a reliability point of view, such studies are particularly useful for clarifying potential interactions in the power system and potential sources of congestion that are not yet verified in real-time operation. In NERC planning-related reliability standards, planning studies are used to analytically demonstrate that the transmission grid can be planned and operated in a reliable fashion. These studies often include pre-commercial studies that analyze and quantify the effects of new generation resources on power flow, stability margins and other reliability parameters that are related to these standards. While the specific analytical tools and methods used in these studies are not described in granular detail in the reliability standards, it is generally considered that the scope of the studies must be such that it addresses material reliability consequences. RTO and ISO interconnection processes outline additional detail that may be addressed in Pre-Commercial Studies. These processes are used to determine the impacts of large numbers of new resources on regional transmission systems, and to ensure that the interconnection studies are done in a similar manner on all projects. From an RO perspective, the regional studies help to ensure that the local conditions in each system as well as the interconnection specifics are accounted for in a coordinated fashion in addition to ensuring that the basic requirements of NERC planning standards are met. Resource modeling within the Pre-Commercial Operation studies is heavily dependent on a number of assumptions in relation to resource capability and performance behaviour. These could include expected operating and operating scenarios, as well as the expected control behavior during both normal and contingency scenarios. Reliability focus has now been on the governance of these assumptions as relates to adequate documentation, scrutiny and

alignment with standard practices associated with grid planning, in order to protect against potential reliability risks that could arise from any misaligned assumptions that are ultimately proved out when the resources come into commission. These pre-commercial studies give a standard against which the performance of later commercial systems can be measured. It gives insight into the likely interactions of the system and its impact. Thus it will serve as a yardstick in determining whether performance of a later system does or does not deviate from expected standards. This baseline-serving purpose is an important instance of continuity between planning evaluations and operating experience. From a reliability perspective, pre-commercial operation studies are not considered guarantees of future performance. Rather, they are best-effort analyses based on the available data and assumptions as of the time the studies are performed. Reliability oversight focuses on whether the scope and analysis of these studies are sufficient and appropriately considered in planning and interconnection processes. Reliability Validation Using Pre-Commercial Operation Studies This chapter examined the use of pre-commercial operation studies as reliability validation tools in the context of the NERC, RTO and ISO standards. In these studies, particular care must be exercised with respect to modeling system interactions, assumptions and planning so that the results of the studies facilitate reliable operation of the new generation while not prescribing any specific engineering implementation or operation practice.

End-of-Chapter Summary

The studies described as Pre-Commercial Operation are really simply a methodized manner to determine, prior to commissioning a new resource to commercial operation, the interfaces between the resource and the Bulk Electric System (BES) as defined in the planning and interconnection processes. Pre-Commercial Operation studies are required or desired by NERC planning standards and RTO/ISO processes and allow a reliability conclusion to be reached regarding the resource prior to commercial operation.

Documented assumptions relating to system interaction and the basic assumptions of the bases for the high voltage component are included in the Pre-Commercial studies in order to ensure the reliability oversight. The Studies do not in any way guarantee performance of the High Voltage Component at a later stage nor impose any technological solution in relation to the design thereof. The Chapter has confirmed that the PreCommercial studies form an integral part of the reliability documentation set within the generation integration lifecycle.

FROM THE FIELD

Pre-COD studies model the project's expected interaction with the system. The model is only as good as the input data, and the data sources matter.

A pre-COD study that didn't model the actual control settings of the inverter is a study modeling the wrong project. Findings on this issue are now common.

Chapter 3

Post-Commercial Operation Studies and Ongoing Reliability Alignment

Studies performed after a generation resource has entered commercial operation are for a different reliability purpose than those performed prior to commercial operation during the planning and interconnection phases. Pre-commercial studies are based on assumptions about future operations. Post-commercial studies of system interactions can be used to test and validate assumptions of resource behavior based on actual system conditions that are known after a unit has entered commercial operation. These studies help to ensure that planning assumptions are aligned with actual system operations. Post-commercial operation studies are carried out following some change in the system, such as: a) A change in system configuration that has occurred, for example, following a planned or spontaneous circuit switch off; b) Differences in the performance of different plants or systems when compared to the previous pre-commercial studies; or c) Changes to reliability criteria and concerns. These studies may be required as a result of changes in the system configuration, changes in generation dispatch, changes to operating limits or a number of other system developments. From a reliability perspective, they provide assurance that the previously assessed interactions in the system still occur and are still relevant to the current configuration and reliability requirements of the system. NERC’s Reliability Standards focus on the power system as a dynamic entity, acknowledging that operational and planning conditions will change over time. When performing analyses to determine system reliability, utilities must consider current system conditions and their projected future state. Similarly, utilities must perform post-commercial analyses to obtain more current knowledge of how their in-service generation resources will interact with the Bulk Electric System given current system conditions. Utilities are not required to document in great detail the activities performed during these studies as long as they are in compliance with the general principles set forth above. Regional Transmission Organizations (RTOs) and Independent System Operators (ISOs) consider the post-commercial study expectations in their operational planning processes, including seasonal look

aheads and monthly system reviews. These studies verify that the new generation resources can be operated within the bounds of reliability, as the conditions of the power system change over time. From an oversight perspective, post-commercial studies help to confirm that reliability evaluations do not end with the interconnection approval of a resource, but continue while the resource is in commercial operation. One of the key reliability concerns in post-commercial studies is to compare assumed with

observed behaviour. No consequential action is required because assumed system performance is not a specification nor a design requirement. It is therefore only an awareness and potential analytical tool. Reliability awareness concerns flagging these occurrences for consideration. Post-commercial operation knowledge is also used for situational awareness across functional boundaries. This helps to keep planners, operators and regulators up to the minute with the latest analysis, enabling them to share a common understanding of system performance. This common understanding of system performance supports better informed operational and planning decisions and prevents outdated assumptions from becoming embedded in operational procedures or plans. Post-commercial reliability studies reaffirm the concept that reliability validation is an on-going activity. From a reliability governance point of view, post commercial studies remind us that our grid is an on-going system that continues to change. Our grid resources operate in a different world every year - a world of growing loads, changes in resource mix and operating practices. The purpose of post-commercial reliability studies is to consider the effects of the new grid conditions and to offer scenarios that may give insight as to the grid operations for the next year, without imposing any mandatory actions on generators. This chapter has detailed the function of post-commercial operation studies to ensure that the behaviour of the generation resources is in line with the BES reliability targets after carrying out the required checks for reliability. It focused on the re verification, on the analysis of the real behavior (based on the observed performance) and on the changing BES conditions with the purpose of maintaining reliability awareness while at the same time avoiding any interference with the commercial operation activity and directly related implementation tasks.

End-of-Chapter Summary

Post-commercial operation studies are required to ensure R1 continuity and to re-validate the relationships between operating resources and the Bulk Electric System in their current in-service configuration. These studies are required under NERC reliability standards as well as by RTOs/ISOs. They occur after assumptions contained in R1 studies are changed.

VGR-3 focuses on re-assessment and comparison of the assumed versus observed plant behavior. Post commercial VGRs are an effective tool to show that reliability validation is an on-going activity. This chapter focused on the VGRs’ contribution to maintaining situational awareness and system-wide reliability, without suggesting specific engineering or operational actions.

FROM THE FIELD

Post-COD studies verify what the project is actually doing. The verification matters because operating performance often diverges from the planning assumption.

Disturbance reports become a major input to post-COD analysis. A disturbance that the project didn't ride through gets investigated, and the investigation often reveals that the pre-COD study didn't predict the actual behavior.

Chapter 4

Interfaces Between NERC Standards and RTO and ISO Study Requirements

Engineering studies for generation resources sit at the intersection of national reliability requirements and regional planning and operations practices. NERC sets the reliability requirements through its standards and related documents, while the RTOs/ISOs dictate the studies required to support that planning and operations. Understanding how these requirements interact with each other is critical to understanding how engineering studies serve as reliability products, rather than simply being a compliance activity. Based on the NERC reliability standards, the Reliability Coordinator (RC), hosts, and other entities must plan and operate the Bulk Electric System (BES) in a reliable manner for a variety of possible scenarios. The standards are often outcome based, and do not prescribe which tools or methods to use for a particular analysis, nor do they specifically define the format of the analyses to be performed. The basic criteria are that the necessary analyses must be available to demonstrate that the Reliability Criteria for System Adequacy and Operating Limit Ratings have been considered. Engineering studies are used as supporting analyses to validate the decisions and planning that are made relative to available generation resources. RTO and ISO study requirements set out to transform the reliability objectives into regional studies that are applicable to the particular system or circumstances. Local topology, operating procedures and planning horizons are not addressed in the NERC reliability standards, and region wide study processes are needed from a reliability perspective to augment the NERC requirements with relevant system information unique to a particular RTO region or ISO market footprint. The relationship between NERC Reliability Standards and RTO/ISO requirements is one of coordination (versus duplication) based on principles of alignment. NERC Reliability Standards specify the reliability effects that must be provided, while RTOs and ISOs direct the structure of studies to determine the basis for assessing those reliability effects within their regions. Reliability analysis focuses on whether the necessary coordination has been established and whether the collection of study processes used by RTOs/ISOs across interconnected regions facilitate consistent reliability analysis. The differences in regional study requirements do not imply any inconsistency in respect of reliability expectations. The differences are due to differences in system design, generation mix and operating practices. Reliability monitoring primarily focuses on whether the studies deal with material reliability implications of the

generation resources. The focus is on the adequacy of the scope and relevance to reliability goals rather than on the methodology. The coordination between the NERC and RTO/ISO reliability planning and operations frameworks is critical where generation resources are required to provide reliability services across multiple timeframes. The generation resources may be required to operate in the short term within the RTO/ISO or planning horizons (medium to long term) as defined by the NERC reliability planning and operations standards. The regional pre-commercial and post-commercial studies contribute to the higher-level reliability analysis used in the RTO/ISO/ISO planning and operations frameworks that are tied to the NERC framework. Reliability analysis views the contribution of these studies as a whole to form a complete picture of the behavior of the system rather than in isolation. Hybrid interfaces can also be an area of high documentation. In fact, the engineering studies carried out on a component can be used for different purposes in relation to the various oversight bodies. In any case, it is important to know the assumptions, limits and conditions under which these studies were made. The reliability oversight does not impose a specific format, but a minimum level of documentation to allow for understanding the correspondence between the results of the studies and the reliability targets. This chapter examined the interfaces of NERC reliability standards and RTO and ISO study requirements with the goal of highlighting their role in enabling reliable integration and operation of the generation resources. Through this examination, the engineering studies can be viewed as coordinated elements within a layered reliability framework as opposed to disjointed or redundant obligations.

End-of-Chapter Summary

The scope of engineering studies associated with generation resources falls at the boundary between NERC reliability standards and RTO/ISO study requirements. NERC sets reliability standards in an outcome-based format, while the regional processes for establishing study requirements embody the unique characteristics of each region, including generation and transmission configuration, operating practices, etc.

Reliability review is concerned with the consistency of these frameworks ensuring that the scope is adequate, the method is transparent and that the method and tools selected are relevant to the reliability objective being considered. This chapter described how, collectively, the coordinated study frameworks support the overall system reliability review.

FROM THE FIELD

NERC sets the reliability requirements. The RTO/ISO administers the study process. Both have authority. The project owner has to satisfy both.

A study that meets RTO requirements but misses NERC requirements is not a complete study. The owner is accountable for the gap, even when the RTO didn't flag it.

Chapter 5

Study Assumptions, Modeling Boundaries, and Reliability Risk

The basis for performing engineering studies for generation resources is determined by assumptions and boundaries, which in turn affect the modeling of the system and the assessment of generation resource capabilities. Within the context of reliability, these assumptions are not trivial; they have a profound effect on the reliability conclusions drawn from the analysis results. Reliability oversight primarily centers on assumptions as to their formulation, documentation and interpretation, rather than on the methodologies employed in the analyses. Assumptions are the conditions under which the validity of the conclusions in a study are maintained. Assumptions may be related to the system topology, to the dispatch procedures, to the load, to the available resources, to the operating conditions etc. In the reliability field assumptions are needed in order to have a systematic approach to analyze a very complex system; but they always represent a source of unknown variability because they deal with real events (system behavior) occurring in conditions not contemplated by the assumptions of the model. These boundaries are included in the engineering model to delimit the scope of the study and as such they define the system, interactions and time frames included or excluded. Such boundaries are defined in respect of planning jurisdiction, study objective, etc. Reliability studies do not need to consider all possible interfacing boundaries, but it is essential to ensure that the chosen boundaries are adequate for determining material reliabilityrelated issues for a generation resource. A common source of reliability risk is assumption vs. reality mismatch. Studies are only valid for so long as the assumptions they were built on are valid. If too much time passes since the studies were done, or if there has been a significant change in operating practice, then the assumptions may no longer be valid, giving a misleading impression of reliability or masking new reliabilty concerns. Regular reviews of assumptions and awareness of the validity of studies over time is important. The difference between planning assumptions and real-time operating conditions is particularly noticeable with generation resources. For example, the pre-commercial studies are based on assumed

conditions and potential generation resources behavior, while the post-commercial studies are based on revised and actual data. Reliability Operations views the studies as a complement to each other, and it is not feasible for all possible operating scenarios to be fully analyzed in a single study in terms of time and operating conditions. Transparency is a reliability concern associated with assumptions and boundaries. Assumptions and boundaries need to be transparent to enabling relevant stakeholders to properly

interpret the results of the study. Transparency does not imply that all assumptions and boundaries should be the same; it implies that there should be consistency in how assumptions and boundaries are interpreted in the context of reliability objectives. Reliability oversight may also cover the reuse of assumptions between analyses, and consideration of the potential consequences of such reuse. Assumptions may be reused without reviewing them for new scenarios, particularly in a dynamic system. Awareness of potential issues with reusing assumptions, as part of a broader reliability oversight effort, is not typically a requirement for strict compliance with assumptions for any particular analysis, nor for adherence to particular modeling approaches. This chapter examined the influence of assumptions and modelling boundaries on reliability values that are derived from engineering studies for generation resources. It has been shown that, by treating assumptions as components of reliability interpretation rather than as merely technical objects, a reliability framework provides tools for the evaluation of an engineering study in a way that does not constrain the methodology used for deriving results.

End-of-Chapter Summary

Engineering study assumptions and modeling boundaries are central to determining reliability conclusions for generation resources. Assumptions are required for analytical analysis, but introduce uncertainty and risk to the reliability of the system if the assumptions do not accurately reflect the true conditions of the system.

Reliability assessment is focusing on the validity of assumptions and bounds as opposed to the methods that utilize them. This chapter demonstrates the impact that assumptions and bounds have on the reliability assessment and argument, and why it is crucial to review them to validate the reliability argument.

FROM THE FIELD

The assumptions that go into a study determine its conclusions. Wrong assumptions produce wrong conclusions, and the wrong conclusions live in the program until the next disturbance reveals them.

Boundary conditions matter. A study that bounded the model too narrowly missed the contingencies that mattered.

Risk assessment is part of the study, not a footnote. Programs that treat risk as a sensitivity analysis have not done the assessment the framework expects.

Chapter 6

Timing, Triggers, and Study Lifecycle Considerations

The timing and the circumstances under which engineering studies are performed and initiated has a direct impact on their reliability value. In the context of the Bulk Electric System (BES) reliability standard, having a study at a particular time and location is not a one-time requirement, but rather a dynamic function that extends throughout the full lifecycle of the planning activities. In this context, reliability monitoring and oversight involves understanding when engineering studies are performed and under what conditions. The pre-commercial operation studies are triggered by proposed changes to the system most frequently related to addition or modification of generation resources. These triggers are based on potential reliability impacts prior to actually adding new equipment to the system. From a reliability perspective, the value of performing these studies is enhanced by proper timing relative to key milestones in the planning process, such as the timing of interconnection approval or initial synchronization. Studies that are performed too early may be based on concepts and assumptions that have yet to mature. Studies that are performed too late may not provide sufficient time for dealing with identified reliability impacts. Post-commercial operation studies are undertaken for various reasons. The reason for undertaking the studies may be a change in system configuration, observed performance departing from expected performance, change in operating constraints or for re-evaluation of the long term system development plan. Reliability supervision considers that not all changes need to be studied in detail. Rather, Reliability supervision ensures that means exist for determining when a review is necessary. Having well-defined criteria for determining when studies are required allows for effective reliability management without introducing undue rigidity or constraints. Study Lifecycle – a simple, yet powerful way of thinking It is important to remember that engineering studies only provide a “picture” of the system at a specific time in the future. As time passes and conditions change, the study will no longer accurately depict system behaviour. This means that rather than being about “adhering” to prior reliability studies, the Reliability engineer should be more concerned with: • The age of prior reliability studies • Their scope and applicable conditions • Their validity and relevance to the current

situation. Reliability Considerations for Studies Lifecycle Coordination refers to the coordination between studies performed throughout the whole lifecycle of a precommercial project and potentially post commercial. In fact, it is often observed that old studies are used for new re-evaluations of a project. Reliability oversight is aimed to confirm that there is adequate continuity in the use of study results and

that changes in assumptions and model variables as time passes are recognised and addressed properly. This continuity is essential to ensure that reliability assessments that extend beyond the short term (i.e., from planning to operational phase and vice versa) are done on a consistent basis. The timing of reliability studies can impact their usefulness to management and oversight personnel. Studies that are synchronized with established planning cycles, seasonal surveys, or known maintenance or other system modifications are more likely to provide relevant information. It is not that reliability oversight requires a particular schedule – rather that the timing of studies and results is suitable for utilization in management of the electric power delivery system in a meaningful way. This chapter so far has explored the impact of time, events and characteristics of component life cycles when assessing the reliability value of engineering studies for generation resources. However, if one looks at reliability studies in more detail as dynamic reliability components, as opposed to fixed specification items, the reliability framework enables dynamic evaluation within a clearly defined boundary of what is technical and what is not – design versus design detail.

End-of-Chapter Summary

The reliability value of engineering studies is not only determined by their quality, but also by the timing at which they are done and the triggering conditions. Thus, pre-commercial studies enable the reliability evaluation of potential system changes before the modifications are actually made, while post commercial studies allow reevaluation of the reliability over time as operating and environmental conditions change.

Engineer les études d’ingénierie comme des actes nécessitant une actualité continue, un suivi et adaptabilité. Reliability oversight considers engineering studies as part of a continuum rather than as a one-off deliverable. This chapter shows how validating the reliability of products, in-line with the timing and the stages of the product lifecycle, allows for a proactive reliability validation without imposing timing and procedures for implementation.

FROM THE FIELD

Verification: did the model run correctly? Validation: did the model match reality? Both have to be performed, and both have to be evidenced.

A model validated against real performance is the discipline that distinguishes reliability work from engineering exercises.

Documentation discipline ages. Studies done five years ago may not match today's record-keeping expectations. Programs have to keep currency or accept the gap.

Chapter 7

Reliability Oversight Use of Engineering Study Outputs

The engineering studies of the generation resources constitute a very important set of analyses that through a structured evaluation enable the reliability analyst to obtain a clearer picture of the reliability risks at the system level. The execution of these studies is purely technical, whereas reliability oversight is related with the planning, implementation and use of the outcomes of these studies to ensure compliance with operational practices and to identify reliability risks, excluding therefore the technical aspects. The aim is to preserve the non-prescriptive characteristic of the reliability model, while enabling an appropriate evaluation of reliability performance. Reliability oversight organizations utilize outputs of engineering studies to determine whether reliability has been considered uniform for all types of generation resources and for all system conditions. The results of these studies are then combined with other information related to power flow, stability margins and system response assumptions. The primary role of oversight is to determine that the conclusions reached in these studies are internally consistent, reasonable and confirm reliability requirements for the power system. Oversight does not verify technical accuracy of any of the individual calculations but rather it validates the logic of conclusions drawn from the studies and their consistency with the reliability requirements for the power system. This function involves comparing results from different studies. For example, if results from different sources or studies for similar types of generation resources differ by more than minor factors when compared under the same system assumptions, then there may be be some differences in assumptions, modelling boundaries or system characteristics that require investigation and clarification. This function is viewed as a part of the reliability oversight activity. Results of engineering studies are used to monitor compliance of system planning. Summary of individual results gives insights on the overall impact of new generation additions on power system dynamics over time. Reliability oversight ensures that the assumptions embedded in engineering studies are still valid as the system structure and generation mix evolves over time, enabling utilities to

monitor and anticipate the cumulative effect of individual additions of new generation. OPERATIONAL CONSIDERATIONS - The output of studies provides operational insight to allow an understanding of expected performance of systems under different operational scenarios. Although the output of engineering studies is not typically used for operational transactions, operational models and practices are based on assumptions that are validated by front end planning and post commercial studies, to

ensure that this link exists and that the studies output is appropriately considered in the reliability framework. Reliability oversight uses results of studies to promote learning throughout the industry. Findings from multiple studies may be summarized in technical reports, referenced in reliability literature and discussed within industry forums to highlight trends and emerging reliability concerns. These materials are informational and analytical in nature and are intended to promote industry-wide awareness of reliability issues, rather than to mandate specific technical actions or corrective measures. From an oversight perspective, the key characteristics of the study output that one needs to consider are transparency and traceability. Based on the discussion with IAEA experts and several participating Member States, transparency and traceability are considered essential features of study outputs to carry out efficient and adequate oversight. Transparency allows for a full understanding of the reasoning and assumptions underlying the study findings as well as the scope of the study to facilitate an objective analysis of the study outcomes and avoid any undue extrapolation or misuse of the results beyond the scope of the study. This chapter has so far explored the use of engineering studies to establish the reliability oversight’s understanding of reliability risk in the system. Now with a focus on the principle of reliability governance being about consistency, alignment and integration rather than being prescriptive, the reliability framework utilizes engineering studies as tools to support a wise and responsive reliability governance.

End-of-Chapter Summary

Reliability oversight utilizing engineering analysis uses the results of studies as evidence to understand the impact of potential design and operational factors on systemwide reliability risk. It uses the comparisons of results, the assessment of consistency of findings, and the incorporation of study conclusions into operational planning and decision-making to understand the total impact of reliability risks and to identify evolving issues.

The importance of transparency, traceability, and alignment with reliability objectives was emphasized in order to ensure that outputs of engineering studies could be used for purposes of industry learning and situational awareness development, without mandating specific technical actions. This chapter further reinforced the fact that study results can be treated as reliability artifacts in the context of an adaptive oversight framework.

FROM THE FIELD

The most common study issue: the input data didn't reflect the as-built configuration. Design data and as-built data diverge, and the study based on design data overestimates performance.

Settings drift. The settings used during commissioning often differ from settings in operation. Studies based on commissioning settings don't predict operating reality.

Vendor data isn't always accurate. The standards expect the project owner to verify, not just to accept.

Chapter 8

Distinguishing Reliability Validation from Engineering Approval

One of the key elements of the reliability program is the understanding of the difference between engineering studies for reliability validation purposes versus engineering studies for project approval or design validation purposes. Although the documentation produced for the studies can be the same for purposes of planning, interconnection, operational activities, the reliability program focuses on how the studies are used for the validation of the reliability of the system versus assessing the feasibility of a project or the adequacy of a design. Reliability validation deals with whether a resource has been modeled and analyzed in a manner that supports Bulk Electric System reliability goals. It is a functional analysis related to potential system interactions, assumptions and impacts as opposed to resource design, interconnection criteria and other more technical and commercial aspects related to the generation of electricity. Reliability oversight does not confirm or endorse a resource study finding, determination or assumption as appropriate, optimal or valid. Decisions made in the context of the engineering approval process for RTO and ISO interconnection projects relate to matters of theoretical or hypothetical feasibility and cost responsibility, and readiness to build, none of which are considered within the reliability framework. While a reliability analysis may be done in association with the studies used to obtain these approvals, reliability oversight is a discrete activity focused on ensuring that all potential reliability impacts of the proposals under consideration are understood. The distinction between verification and validation, and between certification and guarantee, is crucial to understanding the roles and limitations of engineering studies in relation to reliability. An engineering study cannot guarantee that a particular generation resource will operate exactly as predicted under all plausible future scenarios. The purpose of an engineering study is to assess reliability-related issues under specified assumptions, taking into account the data available at the time of the study. Reliability verification is used to highlight these assumptions and conditions. One of the root causes of this misunderstanding is the ongoing and easily made distinction between validation and approval. The expectation of accountability and responsibility then gets misplaced. That the necessary engineering studies are approvals rather than evaluations can lead to important changes in system conditions or component and systems behavior not being re-evaluated. Reliability oversight helps to ensure that the engineering studies are considered more as tools to enhance reliability awareness rather than as the means for providing an approval. This distinction allows for flexibility within the reliability framework. As conditions in the system change over time, projects which were authorized based on studies at one time

in the future may interact with the system in a different way than they did at the time the original approval was obtained. Validation of reliability based on post-commercial operation studies permits re examination of these new interchanges while making clear that the original authorization was proper in the circumstances that existed at the time. Maintenance of this boundary serves the objectivity of the oversight on technological boundaries and promotes constructive dialogue between stakeholders. Reliability oversight as such allows the detection of emerging risks and trends, without calling into question the technical soundness of design approval decisions or imposing corrective measures. The reliability framework, as specified here, remains non-prescriptive and focused on learning. This chapter clarifies the distinctions between the uses of engineering studies in connection with reliability validation versus connection with approval processes, thereby enabling appropriate treatment of system risk within the reliability framework while at the same time providing flexibility and incentives within planning, interconnection, and operating activities.

End-of-Chapter Summary

Engineering studies are used for different purposes within planning, interconnection and operational processes. However, reliability monitoring looks at the studies from the perspective of reliability validation (as opposed to approval or certification). Incorrect interpretation of the fact that a study has been completed in certain circumstances can lead to incorrect assumptions about system performance.

Validation and Approval: The Separation Maintains Reliability Awareness and Freedom of Action Throughout the chapter it has been emphasized that validation and approval should be separated within a reliability framework in order to achieve adaptive revalidation, objective surveillance, and ongoing system configuration to system conditions alignment through maintaining a well-defined separation between the validation and approval boundaries.

FROM THE FIELD

The project owner is accountable for the studies regardless of who performed them. Outsourcing the study doesn't outsource the obligation.

A study that becomes the basis for a compliance argument has to hold up under audit-level scrutiny. The bar isn't engineering competence; it's enforcement defensibility.

Self-Reports related to study issues are increasing. Programs that surface their own study gaps have a different enforcement experience than programs that wait for the auditor.

Chapter 9

System Evolution, Generation Characteristics, and Study Relevance

The importance of engineering studies to generation resources will change in accordance with the changing Bulk Electric System (BES). New generation technologies and operating practices may alter interactions between generation and the transmission/distribution system, potentially affecting assumptions in engineering studies. Reliability activities are dynamic and studies must be viewed in the context of ongoing system changes and developments. New generation resources connected to the Bulk Electric System are operating under a wide range of conditions and their control behaviors, disturbance responses and interactions with system conditions are not the same as traditionally modeled and analyzed. While engineering studies can be a valuable analytical tool to examine the performance of the new generation and its effects on the system, such studies are generally based on assumptions and are therefore limited to the specific conditions considered. As the mix of resources in the generation portfolio changes, the cumulative effect of individual resources can become a material reliability issue. Effects identified in studies of individual projects may not fully account for the effects of the many other similar resources that will be added to the system over time. The reliability analysis process can make use of the aggregate results of these studies, as well as system-wide operational observations, to verify that assumptions made for an individual resource are still valid in the context of the changing resource mix and to identify new effects that may require closer examination. A factor that may have some impact on the relevance of this study is system topology and operations. Transmission projects, retirements and changes to transmission system operating conditions such as voltage and reactive limits may have changed the performance of the transmission system since the base case conditions established for this study were determined. Reliability oversight has verified that engineering studies may not be relevant after changes occur to the transmission system and/or the operational parameters of the transmission system and recommended, where possible, that the studies be updated to reflect current transmission system performance

and any change to reliability standards. It is important to know where in time such investigations take place in a system’s evolution. The findings of investigations made under conditions or assumptions which are considered to be no longer valid may still be frequently cited. It is therefore important to consider when findings or results become outdated and what implies the need for reliability oversight to let us know when it is needed. Reliability overseeing thus does not imply that there is a need to fix an exact

time period between succeeding reliability studies or analysis, but rather when results become outdated so that findings or conclusions of reliability studies, investigations, tests etc. should be treated in the same way as we treat any other results in that the results need to be put into context to avoid misinterpretation. Reliability oversight recognizes that engineering studies are not a crystal ball. It is simply impossible to foresee every eventuality or potential interaction or system behaviour. Work that is done prior to the initial commercial operation can do no more than form a hypothesis for the reliability of the station. The operation of the station itself must then supply the necessary data for evaluation of unexpected interactions or behaviours. The study, with all the effort that has been put into it, should not be rendered valueless by a change in assumptions, requiring a large amount of rework before it can be used again. Continuing surveillance of the station and accumulation of new data will therefore be essential. This chapter investigated the impact of the development of complex systems and the resulting changes in technical generations on the engineering studies. With the reliability approach in mind – where studies are regarded as context-bound and time-bound reliability objects – we were able to provide a contextual reliability assessment of the studies that allow for a dynamic interpretation and therefore dynamic adjustment so that the analysis is always aligned with the current reliability situation of the system.

End-of-Chapter Summary

The relevance of various engineering studies is changing due to evolution in generation characteristics, system configuration and operating practices. The changing state of the Bulk Electric System (BES) is eroding the validity of longstanding assumptions that have formed the basis of these studies.

Reliability oversight is focused on the awareness of the applicability of studies and the necessity of reconsideration when design or operational conditions change, without defining the frequency of validation updates. This chapter emphasizes that a valid reliability evaluation cannot be carried out in a vacuum, ignoring the dynamics of the evolving system under study.

FROM THE FIELD

"The RTO accepted it, so we're done" — RTO acceptance and NERC compliance aren't equivalent.

"The vendor signed off on it" — the project owner is accountable. Vendor sign-off doesn't transfer NERC obligation.

"Studies are engineering deliverables, not compliance ones" — they are both. The compliance dimension is real, audit-relevant, and increasingly examined.

Chapter 10

Synthesis and Reliability Perspective on Engineering Studies for Generation Resources

Engineering studies required or expected by NERC and by RTOs and ISOs are a layer of evidence that underpins the reliability of the Bulk Electric System. These studies facilitate structured analysis of potential interactions between generation resources and the grid on both a planned and real-time basis before and after commercial operation of new resources and technologies. When considered in aggregate, they form a reliability paradigm based on validation, situational awareness and adaptability rather than on regulatory approval or compliance. One of the common threads running through all stages of the study process is the role of engineering studies as validation of reliability assumptions. By applying the appropriate assumptions for the time period under consideration, engineering studies transform the reliability targets into system-level performance evaluations. Reliability surveillance is then more a matter of knowing where, what and when the engineering studies have been developed, their coverage and extent, than with the detailed methods used for their preparation. This object oriented approach to reliability ensures that the methodology used for the reliability analysis is technology independent and is therefore a relevant tool for dealing with the system risk associated with the integration of new generation technologies. Another element that reinforces the dynamic nature of reliability validation is the differentiation between pre-commercial and post-commercial studies. While pre-commercial studies are based on forecasts of expected conditions and proposed resource characteristics, post-commercial studies are planned to allow for evaluation of the validity of those assumptions based on actual performance and real world system conditions. The intent is to establish a continuous process that maintains consistency between assumptions and conditions over time, without constituting a guarantee of future performance. The interfaces between the NERC Reliability Standards and the RTO and ISO study processes also impact the role of engineering studies in the reliability effort. NERC established the reliability analysis criteria to determine the reliability impacts of

the plans and operational procedures developed by the transmission organization. The RTOs and ISOs determine the structure and composition of the studies to fit the local characteristics of their transmission systems. The reliability oversight focuses on ensuring that the processes as a whole are cohesive, transparent, and sufficiently comprehensive, rather than on ensuring that the processes are

the same. Assumptions, modeling boundaries, time frames, and triggers for reliability studies were determined to be important reliability considerations in this study. These factors impact the validity and period of applicability of reliability studies. The reliability assessment process recognizes that reliability risk is comprised not only of analytical determinations, but also of the assumptions, modeling boundaries and time frames associated therewith and the impact of any misalignment therewith due to changes in system operations or design. A clear separation of reliability surveillance versus engineering discretion has been consistently maintained. Engineering assessments, analyses and determinations are not Approvals, Certifications nor Instructions. They are inputs to analytical activities used to gain understanding of reliability aspects of overall system performance. Maintaining this separation has enabled reliable framework operations characterized by increased visibility, learning and responsiveness and thereby, preserved the requisite level of planning discretion and engineering judgment. When considered together, engineering studies for generation resources provide reliability programs with a useful perspective on how planned and actual system operations interact with design and real-time system developments, reflecting the interplay of planned and actual system performance. These studies provide a framework of evidence of the evaluation, reevaluation and context updating of reliability issues over time, and do not pretend to offer certainty regarding any specific future event. Reliability Enhancement Plan Activities: Chapter 9 examines the role of engineering studies as they relate to reliability. As non-prescriptive validation tools to enable an adaptive and informed reliability governance process for generation resources connected to the BES, their role in this context is described in detail.

End-of-Chapter Summary

The engineering studies required or expected by NERC and RTOs and ISOs provide the reliability basis to understand how generation resources interact within the Bulk Electric System across a wide range of time scales – from long term planning to real time operations. They are intended to be used collectively for purposes of alignment, revision and interpretation rather than for strict compliance or design approval.

Outcome-based criteria for transparency of assumptions and for adaptiveness to evolving systems, together with reliability assurance through visibility of engineering studies, provide a framework for reliable reliability monitoring. This chapter brings together the various elements of reliability, as identified in this report, and addresses reliability aspects of engineering studies for generation resources.

Glossary

Glossary

The terms listed below are taken directly from the NERC Glossary of Terms and are used in the context of this publication only.

Bulk Electric System (BES) - Except as modified by the lists below, all Transmission Elements that are operated at 100 kV or higher volts and all Real Power and Reactive Power resources that are interconnected at 100 kV or higher volts. Note: This definition does not include facilities used for local distribution of electric energy.

Commercial Operation - The operational status of a power generation unit or facility that has successfully concluded its trial/commercial testing and commissioning activities and is capable of delivering electricity to the grid in accordance with agreed metering and operating criteria.

Generator Owner (GO) - The entity that owns and maintains generating units.

In connection with the power system, a Generator Operator (GOP) refers to the entity that operates generating units and performs the functions of supplying energy.

Interconnection The physical connection of facilities to the Bulk Electric System for the purpose of transmitting or receiving electric energy.

Planning Authority (PA) The body that ensures co-ordination and consistency among transmission system and service plans, resource plans and protection systems.

Transmission Planner (TP) - The entity responsible for planning the Bulk Electric System in its portion of the Planning Authority area.

This glossary is derived from selected terms found in the NERC Glossary of Terms and is provided as a convenience to users. It does not supersede or modify in any way the official NERC Glossary of Terms.

About the Author

About the Author

Rob Smith is a senior electric industry professional with over thirty years of experience across every major function of the North American Bulk Electric System. His work spans reliability coordination, transmission operations, regulatory compliance, and cybersecurity reliability.

Rob has worked directly in real-time grid operations as a Reliability Coordinator, Transmission Operator, and Power System Operator within RTO/ISO and utility control center environments. He has also held senior regulatory and oversight roles, including senior compliance auditor and subject matter expert for NERC Reliability Standards. In those roles he audited grid facilities for compliance with applicable standards, evaluated the adequacy of mitigation actions, supported the development of violation notifications and settlements as part of FERC-directed enforcement actions, and participated in risk based oversight of utility mitigation activities.

Rob founded Energy Compliance, Inc. to bring senior, regulator-side compliance authority to registered entities directly, without the layered staffing, billable-hour overhead, and generalist advice typical of larger consulting firms. Every Energy Compliance engagement is led by Rob personally.

About Energy Compliance, Inc.

About Energy Compliance, Inc.

Energy Compliance, Inc. is an independent consulting and advisory firm focused exclusively on electric reliability, cybersecurity reliability, and regulatory compliance for organizations connected to the North American Bulk Electric System.

Our work supports registered entities, including Generator Owners and Operators, Transmission Owners and Operators, Reliability Coordinators, Balancing Authorities, and Distribution Providers. We work across NERC Reliability Standards, FERC orders, RTO/ISO market participation rules, Regional Entity oversight, and state regulatory frameworks.

We do this work differently than larger consulting firms. Engagements are led by a single senior practitioner with regulator-side experience. We don’t staff for billable hours. We staff for outcomes. Our deliverables are written to be operationally executable and audit-defensible, not to manufacture activity. Where automation can replace manual work, we build the automation. Where senior judgment is required, the senior is in the room.

Energy Compliance is not affiliated with, sponsored by, or endorsed by the North American Electric Reliability Corporation, the Federal Energy Regulatory Commission, or any Regional Entity.

Services Provided

Our services are written to be clearly defensible. Operationally executable in real time. Audit-defensible at compliance review. Every deliverable is structured for the auditor’s question, not the consultant’s binder.

Energy Compliance services include, but are not limited to:

• NERC reliability and compliance advisory support
• Reliability governance and program assessments
• Registration and applicability analysis
• Operational and engineering reliability alignment
• Compliance program design and improvement
• Audit and enforcement support (non-advocacy)
• Mitigation planning and Self-Report development
• Training and executive briefings on reliability frameworks
• Regulator-perspective program reviews

Each engagement is scoped to the entity’s role, function, and bulk system impact.

ENERGY COMPLIANCE PROFESSIONAL REFERENCE

Rigorous Compliance. Defensible Programs. Energy Compliance, Inc. partners with registered entities on the institutional and technical questions that define strong reliability and cybersecurity programs, from classification through audit through enforcement response.

N ERC CO MP LIANC E S ENIO R ADV ISO RY Program support, interpretation, and audit Direct engagement on complex reliability preparation. questions.

I ND USTRY ENGAGEMENT AUD IT D EFENSE Standards development and working-group Notice of Penalty response and settlement participation. posture.

CONNECT WITH US Scan to visit

E N E RGY COMPL IAN CE , IN C. · EC-WP-504 · © 2026 · AL L RIGHTS RES E RV E D