DUE TO THE PROLIFERATION OF INTERNET-OF-THINGS
(IoT)-based technologies in the last several years, digital computing
hardware and software technologies have seen massive performance
improvement. Additionally, these technologies provide lower costs
for comparatively higher computation and storage, more compact
size hardware, and compatibility with a large selection of operating
systems. Furthermore, communication protocols have increased the
penetration of single-board computers in many consumer and indus-
trial applications. This article presents the application of a state-
of-the-art edge computing infrastructure to the electrical power
distribution grid. Electrical power distribution is becoming increas-
ingly complex with the large degree of integration of distributed
energy resources (DERs). The distribution system also experiences
many different undesired events, such as different types of tempo-
rary and permanent faults, loss of measurement data, and cyberat-
tacks. This article highlights a small-scale experimental validation
of edge computing in power distribution automation that can be used
for classifying different faults, detecting anomalies in the grid, mea-
surement data recovery, and other advanced analytics techniques.
Introduction
With a large number of parallel data sources becoming readily
available in a smart grid, data fusion techniques that combine mul-
tiple data sources lie at the heart of smart grid platform integration.
Hierarchical
Distribution
Grid Intelligence
By James Stoupis , Rostan Rodrigues,
Mohammad Razeghi-Jahromi, Amanuel Melese,
and Joemoan I. Xavier
Digital Object Identifier 10.1109/MPE.2023.3288596
Date of current version: 21 August 2023
38 IEEE power & energy magazine 1540-7977/23©2023IEEE september/october 2023
uthorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on September 04,2023 at 04:42:31 UTC from IEEE Xplore. Restrictions apply.
Using Edge
Compute,
Communications,
and IoT
Technologies
IMAGE LICENSED BY INGRAM PUBLISHING
Related to this concept, the authors de-
veloped intelligent applications that re-
side on edge computing device hardware
in the distribution grid. The developed
edge processor platform performs ad-
visory control functions to assist in the
larger goal of providing enhanced grid
resilience and distributing intelligence
and provides a faster response time to sys-
tem anomalies.
Edge Computing and
Communications Basics
Edge computing combines technologies related
to data processing, communications, and analyt-
ics at the grid edge, which is defined as the distri-
bution system between the substation and the end use
customer sites. Edge computing devices (ECDs) are con-
nected to grid field devices, such as pole top reclosers and
switches, meters, line post sensors, and other field devices, via
wired and wireless communications. Each ECD is also capable of
communicating with its peers on the same distribution system, other
edge processors, and providing communication redundancy and applica-
tion coordination. Furthermore, each ECD is capable of communicating upstream
to substation computers, utility control centers, and even the cloud. See Figure 1 for an
example. Multiple communication media can be implemented using the ECDs, including Wi-Fi,
cellular, and radio communication. Utility preference will determine which media are used for the field commu-
nications and for the upstream communications. The deployment of 5G and future advanced cellular platforms
will only enhance the functionality that is capable of being deployed by ECDs.
The edge processor concept entails the merging of edge computing, ubiquitous communications, and advanced
analytics. As shown in Figure 2, the architecture contains layers of communication devices and intelligent appli-
cations. At the outer layer, low-cost edge computing devices (LC-ECDs) gather data from the local field devices
and communicate this data to the medium-cost edge computing devices (MC-ECDs) or to the high-cost edge
computing devices (HC-ECDs). The MC-ECDs contain basic to slightly advanced applications and communicate
and coordinate between each other and with the upstream HC-ECDs. The HC-ECDs contain advanced applica-
tion and management functions and can communicate to the cloud if desired by system designers.
Edge Computing Framework
The proposed embedded framework for the edge processor concept is shown in Figure 3. The framework is
designed to support two operating system variants (Windows/Linux) to accommodate different applications
and commercialization options. Many communication libraries (industrial communication, web server, pro-
tection, and control proprietary software) are generally hosted in the Windows operating systems with separate
september/october 2023 IEEE power & energy magazine 39
uthorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on September 04,2023 at 04:42:31 UTC from IEEE Xplore. Restrictions apply.
The framework is designed to support two operating system variants
(Windows/Linux) to accommodate different applications and
commercialization options.
development toward containerized solutions that are com- repositories contain various tagged versions of container
puter operating system independent. On the other hand, images and the lower level ECDs can pull down the image
novel secure virtual private network (VPN) technologies they require by requesting the device or using configuration-
(OpenVPN, wireguard, and so on), mesh wireless libraries, specific tags.
and machine learning applications are easiest to evaluate A management bus in this architecture implements con-
and implement in Linux. Container engines such as Docker trol functions, and the container image pulls over secure
or Qemu can then be installed to support containerized protocols. The transfer of applications to specific devices
grid applications. is as simple as providing a descriptive text file for image
Some implementations of the ECDs or device network dependencies to pull or for pulling the actual image itself
may have a hierarchical structure. A high-level block dia- from the higher level ECD.
gram of such implementation is shown in Figure 4. This The management bus features a broker communi-
figure shows a multilayer hierarchal architecture with dif- cating over publish-subscribe type message queuing proto-
ferent capability ECDs (HC-ECD, MC-ECD, LC-ECD). cols, such as message queuing telemetry transport (MQTT),
These device networks feature a central or main ECD with advanced message queuing protocol (AMQP), and so on.
management capabilities for pushing applications to individ- Each application supports receiving message commands to
ual devices. A container registry, hosted either on the high- start and stop application processes, receive configuration-
est level ECD or in the cloud, provides image repositories descriptive files coordinating grouping with other devices, and
for the applications hosted on each computing device. These zone configurations. This management interface also manages
pushing updates to deployed appli-
cations, syncing databases at each
Wired Utility individual device for distributed
Network applications, and any updates to
security or public key infrastruc-
Substation Utility Control ture (PKI) technologies for opera-
Computer Center tional needs. The lower-level ECDs
regularly check for updates to indi-
vidual device configuration and
Other Edge Processors for Cloud Server
Redundancy and Coordination zone configuration files exchanged
over a JavaScript Object Notation
(JSON) or equivalent file format.
Meter A separate communication bus
Pole Top Edge Computing Device provides protocols used for com-
Recloser munication between applications on
each device necessary for wider
area or distributed applications.
Line Post Service Transformer The applications of the communi-
Sensor Monitor cation bus could be sensor mea-
surements, status messages, or
other necessary data interchanges.
This communication bus will
Capacitor Solar also include modbus, distributed
Bank Switch Inverter network protocol 3 (DNP3), and
Downstream Wireless Communications other established protocols used
Upstream Wireless Communications for distribution automation. A few
Peer-to-Peer Wireless Communications examples of base applications
for distribution automation are
figure 1. Edge computing device connections. machine learning, data collection
40 IEEE power & energy magazine september/october 2023
uthorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on September 04,2023 at 04:42:31 UTC from IEEE Xplore. Restrictions apply.
RMS RMS
Data Data
Source Source
RMS
Data
Source Streaming
Data Source
MC-ECD MC-ECD
RMS RMS
Data Data
Source Source
MC-ECD HC-ECD HC-ECD MC-ECD
Streaming
Data Source
Streaming
Data Source
RMS
Data
Source MC-ECD
RMS RMS
Data Data
Source Source
figure 2. Proposed edge computing architecture for power distribution applications.
and fusion, database management, and human–machine interac-
tion or web-server-hosted graphical viewing applications. Mgmt. Comm. Grid-Apps Sensors
Figure 5 shows a high-level process and components Bin/Libs Bin/Libs Bin/Libs Bin/Libs
for converting standalone grid applications (software algo-
ECD Container
rithms) into containers and then deploying, managing, delet-
Docker Image
ing, and updating these containers across many ECDs in the
field application. Qemu Docker Engine
Several tools exist that can be used to accomplish the arm/v6/v7/v8 aarch64 amd64 X86_64
containerization and orchestration tasks. In this demonstra- Multi-Arch Selector
tion, Docker Engine was selected to containerize the appli-
Host OS (ARM64/X86_64)
cation due to its simplicity in implementation. However,
there are multiple tools for orchestration, namely Docker-
figure 3. Edge computing hardware and framework setup.
Swarm, Kubernetes (also known as K8S), and K3S (a cer-
tified Kubernetes distribution for resource constrained or
remote locations). Among these, K3S is the simplest and Cloud Resource
least resource-consuming platform for container orchestra-
tion, which is desired for LC-ECDs and MC-ECDs. There-
fore, K3S was chosen to be the container orchestration plat- HC-ECD
form in this demonstration.
Experimental Prototype Design
A fault detection, isolation, and restoration (FDIR) application
scheme, as shown in Figure 6, was considered for validating MC-ECD MC-ECD MC-ECD
the edge processor concept. Depending on the specific needs
of the application, the edge computers can directly commu- Communication Buses
nicate with intelligent electronic devices (IEDs) and sensors AMQP, MQTT, WS, HTTPS
as well as with other ECDs in the hierarchy. The logical sys-
tem in this demonstration consists of two substations with a figure 4. Multilevel control and communications
typical five-recloser loop, including four protection recloser architecture.
september/october 2023 IEEE power & energy magazine 41
uthorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on September 04,2023 at 04:42:31 UTC from IEEE Xplore. Restrictions apply.
Grid Application Grid Application
(FDIR, Volt-Var, Soft-Reclose..)
Containerized Application
(Docker, Kubernetes)
Containerization
Container Orchestration
Orchestration (DockerSwarm, Kubernetes, K3s...)
User Interface
User Interface
(Portainer-GUI, kubectl-CLI)
figure 5. Simplified process for building containerized grid application on edge computing platform.
controllers, one grid-tie switch, multiple loads, voltage and cur- or supervisor ECD. Each logical state represents the state of
rent sensors, three ECDs, and two gateway devices. Gate- the electrical system. More logical states can be added in the
way devices communicate with multiple ECDs for redundancy future development of complex applications.
and backup. For example, recloser controller R2 can be con-
trolled by both ECD1 and ECD3, and recloser controller R3 Demonstration With Wireless
can be controlled by ECD2 and ECD3, and so on. Sensors and IED Devices
The simulation of the FDIR scheme under test was done A demonstration of the FDIR scheme using wireless line sensors
in a JavaScript-based webserver tool called Node-RED. The and commercial protection and control products is discussed
Node-RED software allows creation of logical and commu- here. The demonstration consists of three edge computing
nication nodes that can represent different components in (three HPE EL10) devices, as shown in Figure 8. The hard-
the FDIR scheme and perform intended control operations. ware setup uses actual recloser controllers (RER620) to
Another benefit of the Node-RED system is that it also has emulate recloser R1 and grid tie GT1 from Figure 6. Other
graphical user interface (GUI) tools that can be used for cre- relay devices are modeled by simplified software switches
ating a very interactive simulation demonstration. in Node-RED. Wireless sensors were fabricated by using
The FDIR algorithm is based on the state machine repre- commercially available line sensors (voltage and current)
sented in Figure 7. The state machine runs inside the ECD3 and digital signal processor-based sensor data acquisition and
wireless modules to connect and
share data with respective ECDs.
Wireless MQTT was used as the commu-
Sensor 1
Fault 1 Fault 2
R R R nication protocol to move the data
Substation 1 R1 R2 and commands between ECD
devices, whereas Modbus TCP
Gateway 1 Fault 3 protocol was used to communi-
ECD1 cate with IED RER620. However,
other protocols are also supported
GT1 Grid Tie 1
ECD3 (e.g., DNP3, IEC 61850).
In the setup of Figure 8, the
Gateway 2 actual electrical load is connected to
ECD2 both wireless sensors. The load volt-
Substation 2 R4 R3 age is standard 110 V 60 Hz and a
R R R
Wireless few resistive loads were connected
Sensor 2 to draw sufficient current for the pur-
Switching Device in “Closed” Position pose of the demonstration. The trip
Switching Device in “Open” Position threshold is set to 0.2 A in the super-
visor ECD (ECD2), meaning when
figure 6. Example FDIR scheme for validating proposed edge processor-based the current through any load exceeds
container orchestration. 0.2 A the respective relay should trip.
42 IEEE power & energy magazine september/october 2023
uthorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on September 04,2023 at 04:42:31 UTC from IEEE Xplore. Restrictions apply.
The wireless sensors measure both voltage and current of their via a router. The K3S master and agent tool are installed on
load and send that measurement data to the respective ECD over respective ECDs. Together, the six ECDs form a K3S cluster.
the wireless local network. As soon as the load current crosses Once the master ECD detects all the agent nodes, the setup is
the threshold, the supervisor takes an action and sends the trip ready to run the FDIR application. The container image for
command to the recloser protection relay. The relay opens and its FDIR application is also developed on two different hardware
status is sent back to the supervisor. Later, when the load current platforms, i.e., ARM7 (Raspberry Pi4) and x86 (EL10). Both
goes below 0.2 A, indicating that the fault is cleared, the relay container image versions are then uploaded to the Docker-
then goes back to the closed position. Hub public repository for test purposes. Each image is about
The GUI panel for one of the edge devices used in this 200 MB. The container orchestration tool, K3S, assures that
demonstration is shown in Figure 9. One thing common in all the set performance metrics of the container cluster system
GUI panels (across all ECDs) is a “system diagram” and fault are met all of the time, for example, start and stop of contain-
status indicators. Apart from that, each edge GUI panel shows ers, updating to the latest version, deleting containers, creat-
settings and statuses relevant to their location in the system. ing more instances of a container in the same edge device,
The GUI accepts certain inputs from the user to simulate dif- deploying a specific container on a specific edge device in the
ferent types of faults as well as displays system status, param- network, and so on.
eters, and waveforms. For example, ECD1 shows the status In this demonstration, for simplicity, a command line
of Substation A breaker, Relay 1 and 2, and has GUI buttons interface is used to deploy and monitor the FDIR appli-
with the capability to inject a simulated fault at locations Fault cation. A configuration file is developed that defines the
1 and Fault 2. The demonstration shows the capability to sim- settings for each ECD. The configuration file includes
ulate faults at three different locations in the two-feeder sub- the type of container image that should run on the tar-
station topology. During normal operation, simulated faults get device(s), port mapping for user interface, number of
can be introduced at different loca-
tions in the system via GUI buttons
While No Fault Detected
for faults. Depending on the fault
type, the algorithm in the ECD Normal
decides to open or close certain Fault Injected
Fault Cleared
recloser devices. The system main- (Manually)
(Manually)
tains the state as long as the fault
exists. Upon removal of the simu- While Fault Power
Not Cleared Fault Fault Type
lated fault, the system is restored to Restoration Identification (Fault 1/Fault 2/Fault 3)
normal operation. Prefault load on Trip/No-Trip GT
faulted circuits is used to check the Immediately After
load before restoration. Immediately After
Fault Fault Identification
This simple demonstration veri- Fault Isolation
Isolation
fies many different aspects of the Trip Relays
edge computing platform, such as B1/R1/R2
connecting to and collecting data
from wireless sensors, connectiv- figure 7. Simplified state-machine algorithm for FDIR scheme.
ity with existing protection relay
devices, as well as distributed com- Wireless Sensor 1 Wireless Sensor 2
puting and relaxed real-time con-
trol through the edge computing 110 V, 60 Hz 110 V, 60 Hz
architecture. + Load + Load
Demonstration With
Software Containers and ECD3
Orchestration EL10
MQTT
A final demonstration for FDIR ECD1
using container orchestration is EL10 LAN SW
Modbus
discussed here. The demonstration TCP
Wi-Fi
ECD2
consists of six ECDs (3 × EL10 and Router RER620 RER620
EL10
3 × Raspberry Pi4) as shown in Fig-
ure 10. All the edge devices are con- Recloser 1 Grid-Tie Relay
nected over a wireless network that
is also connected to the Internet figure 8. Edge processor hardware setup for validating FDIR scheme.
september/october 2023 IEEE power & energy magazine 43
uthorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on September 04,2023 at 04:42:31 UTC from IEEE Xplore. Restrictions apply.
