Cognitive Robotics 2 (2022) 132–146

Contents lists available at ScienceDirect

Cognitive Robotics
journal homepage: http://www.keaipublishing.com/en/journals/cognitive-robotics/

Survey of emotion recognition methods using EEG information

a r t i c l e i n f o a b s t r a c t

Keywords: Emotion is an indispensable part of human emotion, which affects human normal physiological
Artificial intelligence activities and daily life decisions. Human emotion recognition is a critical technology in artificial
Electroencephalograph signals intelligence, human-computer interaction, and other fields. The brain is the information process-
Preprocessing
ing and control center of the human body. Electroencephalogram (EEG) physiological signals are
Feature extraction
generated directly by the central nervous system, closely related to human emotions. Therefore,
Emotion classification
EEG signals can objectively and now reflect the human emotional state in real-time. In recent
years, with the development of the brain-computer interface, the acquisition and analysis tech-
nology of human EEG signals has become increasingly mature, so more and more researchers use
the research method based on EEG signals to study emotion recognition. EEG processing plays a
vital role in emotion recognition. This paper presents a recent research report on emotion recog-
nition. This paper introduces the related analysis methods and research contents from the aspects
of emotion induction, EEG preprocessing, feature extraction, and emotion classification and com-
pares the advantages and disadvantages of these methods. This paper summarizes the problems
existing in current research methods. This paper discusses the research direction of emotion clas-
sification based on EEG information.

1. Introduction

Human emotion is a psychological and physiological state produced by the process of human consciousness, which plays a vital
role in interpersonal communication [1]. Human beings do not start thinking from a blank brain. Emotions are indispensable in
normal physiological activities and decision-making in daily life [2]. Brain activity plays an essential role in emotion generation and
movement, and EEG signals collected from the brain can detect the information related to the change in the emotional state [3,4].
People’s behaviors are closely related to emotions. In most cases, emotions can reflect people’s actual thoughts, emotional states, and
behaviors caused by psychological conditions [5]. EEG signals can reflect people’s emotional state in real-time and objectively. There
are ways to obtain emotions: extracting animated features from text [6], extracting expressive features based on facial expressions
[7], and collecting EEG signals from subjects to extract dynamic segments. In recent years, with the development of dry electrode
technology, non-invasive electrode devices have been widely used in EEG acquisition [8,9].
The development of the brain-computer interface is of great scientific value and significance to the research on emotion recognition
[10] and artificial intelligence [11]. Emotion recognition based on EEG signals has a broad application prospect in entertainment and
games, education, criminal investigation, assisted driving, health care, and other fields [12,13].
For example, we can obtain drivers’ emotional conditions by analyzing their EEG signals in assisted driving. In anger, drivers are
likely to make dangerous driving behaviors, and traffic accidents can quickly happen [14–17]. Cheng Jing [18] predicted that the
average speed of drivers under different emotional states would increase significantly compared with the intermediate state through

∗
Corresponding author.
E-mail address: [email protected] (M. Wang).

https://doi.org/10.1016/j.cogr.2022.06.001
Received 1 May 2022; Received in revised form 20 June 2022; Accepted 22 June 2022
Available online 29 June 2022
2667-2413/© 2022 The Authors. Publishing Services by Elsevier B.V. on behalf of KeAi Communications Co. Ltd. This is an open access article
under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

Fig. 1. The schematic diagram of emotional recognition.

drivers’ ECG signals and driving data. Mingen Zhong [19] believes that anger and road conditions impact drivers’ speeding behavior.
Emotion recognition technology can predict and prevent drivers’ driving behaviors [20]. The emotion recognition process includes
induction, signal acquisition, preprocessing, feature extraction, and classification. The schematic diagram of emotional recognition
is shown in Fig. 1.
This paper presents a recent research report on emotion recognition. This paper introduces the related analysis methods and
research contents from the aspects of emotion induction, EEG preprocessing, feature extraction, and emotion classification and com-
pares the advantages and disadvantages of these methods. This paper summarizes the problems existing in current research methods.
This paper discusses the research direction of emotion classification based on EEG information.
The structure of this paper is as follows: the second part introduces the emotion model and its elicitation. The third part is EEG
signal acquisition and classification. The fourth part summarizes and compares the signal preprocessing methods. The fifth part
summarizes and compares the signal feature extraction methods. The sixth part summarizes and compares the feature classification
methods of EEG signals. Finally, the outlook and summary.

2. Emotion overview

Emotion is a general term for a series of subjective cognitive experiences, a psychological and physical state produced by syn-
thesizing multiple feelings, thoughts, and behaviors. In real life, emotions have complexity and diversity. How to classify human
emotions reasonably is a complex problem in emotion research. The brain’s central nervous system is closely related to the gener-
ation of emotions. The brain’s central nervous system directly generates EEG signals, so EEG signals are directly related to human
emotions.

2.1. Emotional model

Studies show that human beings have hundreds of thousands of emotions, different emotions have different characteristics, and
emotions also have various forms of expression in daily life and work. How to classify emotion is a complex problem in emotion
research. The simple classification is a positive emotion, neutral emotion, and negative emotion. But emotion is often a synthesis of
many basic emotions, a continuous physiological state. So psychologists tend to define emotions in terms of multi-dimensional space.
Schaaff [21] divided emotions into positive, neutral, and negative emotions by using the standard emotional pictures of the
international dynamic picture system as the induction mode. Takahashi [22] divided emotions into five categories: happiness, fear,
disgust, sadness, and relaxation, using video stimulation to induce feelings in 12 experimenters, but the recognition rate was not
high. Panagiotis [23] classified emotions into six emotional states of pleasure, surprise, anger, panic, disgust, and sadness by using a
high-order cross coefficient. Currently, the commonly used emotion dimension is the two-dimensional emotion classification model
proposed by Lang [24], as shown in Fig. 2.
In Fig. 2. The emotional types in the figure are in a two-dimensional discrete pattern and different emotional states in this two-
dimensional space. The horizontal axis, from left to right, boredom to excitement, represents the intensity of the emotional response.

133
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

Fig. 2. Lange two-dimensional emotion classification model.

The vertical axis goes from top to bottom, from sadness to liking. In the model, the spatial distance between different emotions is
directly proportional to the degree of similarity of feelings. The closer the spatial distance is, the more similar the sentiments will be,
and the farther the spatial distance is, the more different the emotions will be.

2.2. Emotions induced

The subjects must be in the emotion required for emotion recognition before the subsequent EEG signal collection. The commonly
used emotional induction methods include the imagination induction method and the event induction method.
In the process of EEG signal collection, the subject imagines the scene of a given emotion according to the hints of the researcher
to achieve the purpose of internal induction. Placidi [25] presented participants with different smells before the experiment and then
asked them to recall the gas at that time to induce specific emotions. Ekman [26] made subjects produce other emotional scenes
through various prompts to obtain corresponding EEG signals.
The image-inducing method has specific requirements for the subject, the external event induction method has low requirements
for the issues, and it mainly induces the corresponding emotional characteristics of the subjects through external stimuli, such as
pictures, music, and videos with specific emotions. Imagination induction has some uncontrollable factors, making it easy to produce
incorrect installation. Therefore, in most cases, controllable event induction is generally used to induce emotions. Currently, the
commonly used method of event induction is the International Emotional Picture System (IAPS) established by Lang [27] and the
International Emotional Audio Library (IADS) based by Bradley [28].

3. EEG signals

EEG signal is a kind of bioelectrical signal, which is the overall response of the activity of a large number of neurons in the cerebral
cortex or the surface layer of the scalp and contains a lot of physiological and pathological information [29,30]. Relevant studies
show that collecting and analyzing EEG signals and extracting their emotional features can reveal necessary information about human
emotional states. Dynamic features can judge the physiological and psychological conditions of human bodies.

3.1. EEG signal acquisition

The generic form of EEG signal can go into spontaneous and induced modes [31]. Spontaneous EEG is the spontaneous physio-
logical activity of brain cells in the human cerebral cortex. The experimenters directly recognize emotions through facial expressions
[32]. Evoked type stimulates brain cells externally with specific visual or auditory stimulation through the experimenter’s neural
pathway to generate EEG signals with corresponding characteristics [33]. In practice, the researchers used evoked expressions to
stimulate subjects to produce electrical signals. EEG signal collection can be either invasive or non-invasive. Due to the high risk
of invasive acquisition on the human body, non-invasive EEG signal in brain-computer interface research due to its portability and
safety. Therefore, researchers adopt non-invasive EEG signal acquisition [34,35]. Non-invasive EEG signal acquisition in Fig. 3.
There are two kinds of electrodes for signal acquisition: dry and wet. The collection method of the dry electrode is to collect the
EEG signal directly by contacting the dry electrode with the scalp without adding any conductive medium in the middle. Due to
the large impedance of the cuticle, the collected EEG signal is relatively weak. The subject will apply a conductive paste to increase
electrical conductivity. In scientific research, the collection method of the wet electrode is more stable and effective. Non-invasive
EEG acquisition can efficiently record and analyze the brain activity signals of subjects [36] and has the advantages of being low cost
and easy to wear. The availability of non-invasive EEG acquisition devices will be significantly improved [37].

134
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

Fig. 3. Non-invasive collection of EEG signals.

Table 1
Characteristic frequencies of different brain waves.

Types Characters

Delta wave (0.5–4 HZ) Delta waves occur in the temporal and parietal lobes and are associated with deep relaxation and restorative sleep in
humans.
Theta wave (4–8 HZ) Theta waves are common in people who are trance or hypnotic. In this state, the brain’s Theta wave is optimal.
Alpha wave (8–14 HZ) Alpha waves occur in the occipital and posterior parietal lobes. When awake, quiet, and eyes closed, the wave
amplitude is like a shuttle pattern from large to small and then from small to large.
Beta wave (14–30 HZ) Beta waves are the most common high-frequency waves seen during wakefulness and occur mainly on both sides of
the brain.
Gamma wave (>30HZ) Gamma waves are essential for learning, memory, and processing and combine sensory tools for processing new
information.

3.2. Classification of EEG signals

EEG signals can into Delta wave, Theta wave, Alpha wave, Beta wave, and Gamma wave by frequency [38]. There is a strong
correlation between EEG signals in a specific frequency range and different brain activity states. The classification of EEG signals is
in Table 1.
Delta wave (0.5–4 HZ): Delta waves to assess sleep depth, and an increase in Delta wave power is associated with increased
attention in internal working memory tasks [39].
Theta wave (4–8 HZ): Theta waves are associated with a wide range of cognitive processing, such as memory encoding and
retrieval. Theta is also associated with increased fatigue [40].
Alpha wave (8–14 HZ): Alpha waves bridge the gap between the conscious and subconscious mind. When the individual brain
frequency is in the alpha wave, people’s consciousness is awake, and the body is relaxed [41]. In this state, physical and mental
energy expenditure is minimal. The alpha wave was highly correlated with valence [42].
Beta wave (14–30 HZ): Beta waves often appear when individuals are in a state of concentration, vigilance, and other active brain
thinking, and the frequency of Beta waves will become stronger [43]. Significantly higher Beta wave frequencies were observed in
the brain when the body was actively active [44]. Beta wave has a high correlation with the excitation state of brain neurons.
Gamma wave (>30 HZ): Gamma waves are related to multimodal perception processing. Studies have shown that the Gamma
wave reflects the concentration of attention [45]. Gamma waves are associated with rapid eye movement [46]. Li [47] used picture
stimuli to induce subjects’ emotions and found that the Gamma band was the best emotional band for most issues, reflecting the
critical role of the Gamma band in emotion recognition research.

135
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

Fig. 4. EEG signal preprocessing methods.

4. Signal data preprocessing

EEG signal is a bioelectrical signal of the brain with 5–100 μ v and low frequency. The signal can be displayed and processed after
amplification by an amplifier. Because of the high sensitivity of EEG signals, it is easy to interfere during the acquisition process. As a
result, the EEG signal collected is weak, and the analysis result of the EEG signal is often unsatisfactory. It brings great difficulties to
the analysis of EEG signals. These interference noises affect EEG signals’ acquisition and analysis [48,49]. EEG signal preprocessing
removes some other noise signals from the EEG signal, such as EOG artifacts, EMG artifacts, ECG artifacts, electrocutaneous response
(GSR), power frequency interference, and electromagnetic interference [50–52]. EOG and EMG can filter the noise by adaptive
filtering, spatial filtering, and blind source analysis. The EEG signal preprocessing methods are in Fig. 4.

4.1. Principal component analysis

Principal Component Analysis (PCA) is a statistical method and a data dimension reduction algorithm widely used in various
fields. Mainly applied in pattern recognition, PCA can use for dimensionality reduction of high-dimensional feature space and reduce
redundant information contained in data. In recent years, PCA can be used to remove EOG artifacts from EEG data in EEG data
preprocessing [53,54]. PCA is used to reduce the dimensionality of the EEG data. Then separate electroophthalmic components are
subtracted from the collected data to minimize the amount of data and computation time required. The basic principle of PCA to
remove EOG is to map the N-dimension features of EEG data to the K-dimension, which is called the principal component. The
K-dimension features are reconstructed based on the original N-dimension features to remove unnecessary noise interference and
realize the separation of EEG data.
S. S. Poorna [55] used the PCA algorithm to eliminate noise interference in EEG data preprocessing. EEG data were collected using
the wearable Emotive EPOC, a device designed to collect 14-channel 128 HZ EEG data. Data from EEG data of 18 healthy adults aged
around 20 years and stimulated by a 2-h 3D driving simulation using a VR box device. The collection began when subjects were about
to enter a fatigued state. Feature extraction using FIR filter. These features classify the driver’s emotions into four categories: alert or
awake, sleepy, and sleep state, which can monitor the driver’s driving condition and reduce accidents. Finally, k-Nearest Neighbor
(KNN) and artificial neural network (ANN) classify these features with the accuracy of 80% and 85%, respectively. The sensitivities
were 33.35% and 58.21%, respectively. The results show that the PCA algorithm effectively reduces data dimension.
Ehtasham Javed [56] proposed a hybrid algorithm combining empirical mode Decomposition (EMD) and principal component
analysis. The algorithm can remove BCG artifacts. The EEG data were collected from 11 subjects with no history of mental illness
in an MRI scanner using the EEG Cap tool. The EEG Cap tool is a 128-channel sensor network. They data into two groups: the first
group of EEG data with eye closure and the other group with induction. The researchers used the EEGLAB tool to eliminate noise
and applied a hybrid algorithm. The results show that this method can effectively retain EEG signals with less loss and remove BCG
artifacts. Compared with other algorithms, this method has better performance.
Md. Asadur Rahman [57] used the PCA algorithm and T-statistics to reduce the dimension of EEG data. The principle of the
method is that the PCA algorithm finds the direction with the most significant variance of the original data as the main feature, and
the PCA algorithm is to nine channels F7, F5, F3, F1, Fz, F2, F4, F6, F8. The experimental data came from 15 groups of data in the
SEED dataset and induced stimuli to stimulate the subjects to generate corresponding emotions using clips of inspirational videos
(relaxation, sadness, and joy). The researchers collected EEG data using an ESI neural scanning system. The EEG data is sampled to
200HZ and then filtered by a 50HZ notch filter and Matlab to remove artifacts. Finally, the researchers apply the EEG data to the PCA

136
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

algorithm. The results show that the classification effect is better. PCA can find alternative variables under minimum information
loss to achieve dimensionality reduction.

4.2. Blind source separation

Blind source separation (BSS) is a powerful processing method in signal processing. The principle is the process of separating
unknown source signals from observed signals. It is in medical signal processing, data mining, speech enhancement, image recognition,
radar, and communication signal processing. BSS can be used to remove EOG and EMG in EEG data processing. The most commonly
used method of BSS is Independent Component Correlation Algorithm (ICA). ICA algorithm is a blind source separation method based
on high-order statistical characteristics of signals [58,59]. ICA was first studied in neural networks and applied to signal separation
and feature extraction. ICA algorithm can separate EOG signals from EEG data [60]. ICA algorithm is a linear transformation process
that separates data or signals into a linear combination of statistically independent non-Gaussian signal sources. The use of the ICA
algorithm needs to meet three conditions [61–63]:

I The components are independent of each other. That’s the basic principle.
II The independent components must be non-Gaussian.
III The unknown mixture matrix A is A square matrix.

Anneleen Vergult [64] proposed a BSS-CCA method to remove EMG artifacts from EEG data. The BSS-CCA method combines
BSS with statistical canonical correlation analysis (CCA) to avoid the shortcomings of the ICA algorithm in removing too few EMG
artifacts or correcting brain activity. EEG data from 37 non-epileptic patients with an average age of 33 years. Emotion from video
stimulation and EEG data by a 21-channel OSG EEG tool. The EEG data were de-frequented with a band-pass filter and de-noised
with a notch filter. The data were evaluated by epileptic specialists using EEG signals recorded about 50 s before an epileptic seizure.
Experimental results show that BSS-CCA is superior to ICA in removing EEG EMG artifacts.
Junfeng Gao [65] proposed an EOG removal method based on ICA and manifold learning. The data came from EEG data collected
from 16 healthy adults, whose EEG signals were generated by blinking symbols on a screen in response to arrows flashing left and
right. After selecting 16-channel EEG signals, The researchers used the EEGLAB toolbox to divide the signals into multiple data sets for
experimental comparison. In this method, the initial features are extracted by processing EEG data with ICA, and then the manifold
learning algorithm is introduced to reduce the dimensionality of the initial components. Finally, the k-nearest neighbor classifier is
to classify the initial features and better classification results, but there are no more ICA and various learning algorithms. The artifact
removal accuracy of BSS is very high, but it is difficult to determine the information and mixing mode of the source signal. The
determination of variance, energy, and independent order of independent components in the ICA algorithm is a problem that needs
to be solved and discussed by researchers.

4.3. Detrended fluctuation analysis

Detrended Fluctuation Analysis, abbreviated as DFA method. This method is applicable in all fields and is a scale signal analysis
method suitable for analyzing the long-range correlation of signals [66]. The researchers can use this method to remove artifacts
from EEG data. The detrending process can eliminate the influence of the offset generated by the sensor when acquiring the data for
the last calculation. That is, The researchers can delete the trend from the data, and the analysis can focus on the fluctuation of the
data trend itself. The principle of DFA is to subtract a least-square fitting line, plane, or surface from the data so that the mean value
of the detrended data is zero.
Chamandeep Kaur [67] proposed a new denoising method combining DFA and wavelet packet transform. The researchers collected
the experimental data from 34 patients with major depression and 30 regular participants of Hospital Universiti Sains Malaysia
(HUSM). The data was sampled and filtered down to 256HZ, and then the denoising method was applied. Firstly, the EEG data were
decomposed into various modal functions by empirical mode decomposition (EMD). DFA as a standard for selecting EEG data, and
finally, clearer EEG signals were extracted based on wavelet packet decomposition. The results show that the accuracy of emotion
classification improves after denoising, but the diagnosis of depression needs to integrate many aspects.
Rachel J. Smith [68] collected EEG signal data from 40 epileptic spasms (ES) patients and 20 normal subjects. The emotion-
inducing method used 19 h of evening video stimulation. The researchers took EEG data of two wakefulness and two sleep periods of
at least 20 min for each participant. The researchers used the Kohden device to collect 200HZ EEG data from 19 standard channels.
DFA was used to compare EEG data from epileptic patients and healthy subjects.
In contrast, the Shannon and permutation entropy of ES patients was lower than that of the control group, and the DFA intercept
value was higher than that of the control group. At the same time, there was no significant difference in the DFA index value. Medical
staff can combine these objective differences in EEG data with other clinical factors to speed up diagnosis and treatment.

4.4. Notch filter method

Filtering is the effective extraction of a signal at a particular frequency and the specific processing (gain, attenuation, filtering) of
the extracted signal. The basic principle of notch filtering is rapidly attenuating the input signal at a particular frequency point to block
the frequency signal through the filtering method. Traditional filtering methods do an excellent job of eliminating high-frequency

137
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

artifacts. Still, they can lead to the loss of EEG components because ECG artifacts have spectral overlap with the underlying EEG
signal [69,70].
Chenxi Dai [71] proposed a recursive least square notch filtering method (RLS) to remove ECG artifacts. The data into two
parts: EEG signals from the mice after suffocation and EEG signals from the mice a few minutes before resuscitation. To ensure
the experiment was accurate, the mice were untreated. The researchers recorded four data channels using a differential preamplifier
instrument, which was by a Windaq system and reduced to 1000 HZ. The simulation on an algorithm designed to simulate EEG signals
from nine Sprague Dawley mice. The simulation results show that compared with ICA and Infomax, RLS can effectively eliminate
ECG artifacts in EEG and retain most EEG information with minimal loss.

4.5. Advantages and disadvantages of EEG signal preprocessing methods

The principal component analysis is for data dimension reduction, which benefits subsequent analysis and data processing. Blind
source separation usually uses independent component analysis (ICA) to separate source signals from linear mixed signals of multiple
sources. The selection of objective function and optimization algorithm are the critical points of the ICA application. Detrended
Fluctuation Analysis is suitable for all kinds of non-steady time-series research, which can effectively avoid the pseudo correlation
interference due to noise and signal instability. Filtering is a common EEG data preprocessing method to reduce baseline drift and
power line noise interference. The stopband of notch filtering is very narrow, and the starting order must be above the second-order
(including the second order). The advantages and disadvantages of the signal data preprocessing methods are in Table 2 below.
Each EEG data preprocessing method has its advantages and disadvantages. In practical application, researchers must select
pretreatment methods according to the actual situation. At present, removing artifacts usually combines two or more techniques and
combines the advantages of each approach to achieve a higher recognition rate of signals.

5. EEG signal feature extraction

EEG signals with less interference can be obtained by preprocessing EEG data. Emotion recognition based on EEG information
needs to extract EEG features that can represent different emotional states from the original EEG signals collected and then classify
emotions according to these EEG features. Therefore, feature extraction is a crucial stage in the process of emotion recognition, and
finding suitable parts is the guarantee of subsequent data processing. EEG signal feature extraction uses EEG signal as the source
signal, determines various parameters, and forms the vector to represent the signal feature. Characteristic parameters include the
time, frequency, and space domain [72,73]. In recent years, in the research of emotion recognition based on EEG information,
Feature extraction methods with sound effects are Adaptive Auto-regressive (AAR), Fast Fourier Transform (FFT), Wavelet Transform
(WT), and Common spatial pattern (CSP).

5.1. Adaptive auto-regressive

Adaptive Auto-regressive (AAR) is a standard feature extraction method for EEG data. Mathematically, the principle of the AAR
model is that the parametric method changes with the input of each local point, which can well reflect the state of the brain and
adapt to the changes in the EEG spectrum.
Li [74] proposed a feature extraction method combining AAR model parameters and cumulative frequency band power. Firstly,
wavelet transform and inverse transform are to decompose and reconstruct EEG data, and the AAR model parameters and cumulative
band power are feature vectors. The data from EEG signals generated by motor imagery stimulation in healthy 25-year-old women.
Participants will gradually focus from the relaxed state according to the cues. In the inducement of on-screen alerts, participants
perform random left and right-hand imagines to move on-screen arrows. The experimental data were into the left and right-hand
training groups and test groups. The researchers reduced the frequency of the data to 128HZ and then used a low-frequency filter
to denoise the data. Finally, The researchers saved the data in file form with Matlab software. The results show that the AAR and
cumulative band power combination is superior to other extraction methods.
Qi Z [75] used the AAR model to extract EEG data features induced by different luminance. The data were from EEG signals
generated when healthy adults were subjected to video stimuli random at four different brightness levels, each repeated several

Table 2
Comparison of advantages and disadvantages of EEG signal preprocessing methods.

Preprocessing Advantages Disadvantages References

PCA Reducing the dimension of data for easy Low precision of signal processing [55–57]
analysis
BSS Fast convergence speed and high precision The calculation process is tedious, and the [64,65]
precision is low
DFA The researchers can quantitatively analyze the Signal processing is inefficient [67,68]
interference in the signal
NF Effective removal of artifacts Some EEG Information is missing and [71]
low efficiency

138
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

times. The participants gradually focused from the relaxed state according to the brightness stimulus. The researchers used HydroCel
instruments to record EEG signals at about 1000 ms of stimulation. The EEG signals were sampled, filtered, and lowered to 0.5–30 HZ.
They built a minimum distance analysis (MDA) classifier to distinguish between different brightness perceptions and successfully
analyzed the various emotions of the subjects for the four types of brightness.

5.2. Fast Fourier transform

Fast Fourier transform (FFT) is a traditional and validated EEG feature extraction technique. FFT is a fast algorithm of discrete
Fourier Transform (DFT). Based on the discrete Fourier transform’s odd, even, virtual and natural characteristics, FFT is by improving
the algorithm of the discrete Fourier transform. It is a promising stationary signal processing tool [76]. The following equation
describes the DFT:
𝑁−1
∑
𝑋 (𝑘 ) = 𝑥(𝑛)𝑊𝑁𝑛𝑘 (𝑘 = 0, 1.....𝑁 − 1) (1)
𝑛=0
𝑁−1
∑
x (𝑘 ) = 𝑋 (𝑛)𝑊𝑁−𝑛𝑘 (𝑛 = 0, 1.....𝑁 − 1) (2)
𝑘=0
2𝜋
𝑊𝑁 = 𝑒−𝑗 𝑁 (3)
Fourier transform is suitable for analyzing the frequency domain signal in the time domain signal. The calculation principle is
obtain the frequency domain signal X(k) from the time-domain signal x(n) 𝑁 represents the total number of discrete signals. In the
formula, x(n) represents the input time-domain signal, and n represents the number of the input time domain signal. X(k) represents
the frequency domain signal output by the DFT formula, and 𝑛 represents the number of the frequency domain signal output. WN is a
rotation factor, and j represents 90° clockwise or counterclockwise rotation. The basic idea of FFT is to use periodicity and symmetry
in DFT to transform the calculation process of DFT into an iterative algorithm, which can significantly improve the efficiency and
accuracy of operation.
Delimayanti M K [77] used FFT to extract features from EEG data. The data came from a subset of the publicly available sleep EDF
dataset, including EEG signals from 42 men and women. The data were into two groups, one set of EEG signals from healthy adults
and the other set of EEG signals from adults with mild sleep difficulties. Data were collected during the subjects’ daily life, using the
international standard 10–20 system to collect EEG signals from both channels. The principle of feature extraction is to divide the EEG
data into the 30 s, process it with the FFT algorithm, select the features with the random forest algorithm, and combine thousands of
FFT features as experiments. The results show that combining high-dimensional FFT features and simple feature selection improves
classification. Esmeralda C [78] improved the accuracy of EEG data from 60% to 80.3% by using Adam’s optimization model and
feature extraction of FFT.

5.3. Wavelet transforms

An EEG signal is a typical non-stationary signal. Wavelet Transform (WT) is a usual multi-resolution time-frequency analysis
method with good resolution and scaling characteristics, which is suitable for analyzing non-stationary signals in the frequency and
time domains [79]. The basic principle of WT is the localized analysis of the time and frequency domain in EEG data and the signal
data at multiple scales through stretching and translation operation. Finally, the time subdivision of high frequency and frequency
subdivision of low frequency. Therefore, as the frequency of EEG signals is by wavelet transform, the frequency decreases gradually
[80,81].
Zhenman Ma [82] proposed a feature extraction method combining wavelet transform and sample entropy (SE), which used
the high resolution of wavelet transform and had high feature extraction accuracy. The data were from EEG signals generated by
motor imagery stimulation in healthy subjects. The subjects performed left and right-hand motion imagination according to random
prompts, and each cue lasted about 4S. The experimental data were into test and training groups, and each group’s data included
nearly 30 left and right-hand motion imaginations. The EEG data were denoised using wavelet transform, decomposed by MATLAB
multilayer, feature selection based on SE, and finally classified by linear discriminant analysis. Experimental results show that this
method has high precision.
Xiaoqing Zhu [83] proposed a feature extraction method combining an autoregressive model and wavelet transform. The exper-
imental data came from the open data set "BCI Competition 2003″. The researchers collected the left and right imagination signals
from a healthy subject whose subjects gave a random left and right arrow cue. Each cue lasted about 9 s. There were 70 sets of training
data and 70 sets of test data, each set of 9 s. The researchers first used an adaptive autoregression model and wavelet transforms for
feature extraction, then used a genetic algorithm for optimal feature selection, which GA Toolbox realized in Matlab, and finally used
a support vector machine (SVM) for recognition and classification.
Y Hu [84] proposed an EEG data extraction algorithm based on WT and STFT. The experimental data came from the open data
set "BCI Competition 2003″. The EEG signals come from the subjects’ motor imagery, and the brain centers generate different EEG
signals based on the subjects’ additional arm imagery. The experiment was in a win10 environment, TensorFlow built the algorithm,
and MATLAB realized the data visualization. The researchers first perform wavelet transform on the data, then extract features using
the STFT algorithm, visualize the EEG features as the input of the convolutional neural network, and the classification results are by
the Softmax function.

139
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

5.4. Common spatial pattern

Common Spatial Pattern (CSP) is a spatial filtering feature extraction algorithm for two-classification tasks, which can extract
spatial distribution components of each category from multi-channel EEG data and effectively extract frequency band variance as
features [85].
The basic principle of the CSP algorithm is to use diagonalization of the matrix to find a set of optimal spatial filters for projection,
maximize the variance difference between the two kinds of signals, and obtain feature vectors with high discrimination. CSP is highly
dependent on same-based covariance, and multi-channel is very sensitive to noise and is prone to over-fitting in small sample Settings
(SSS) [86]. Traditional EEG feature extraction methods based on frequency feature only extract the energy feature of each channel.
Tang [87] proposed a feature extraction method of B-CSP. The data were into two groups. The researcher collected the first data
set from nine healthy subjects’ EEG signals evoked by visual stimuli. The subjects imagined moving left and right hands according to
the cross icon on the screen, which turned the ICONS into left and right arrows. After collecting, the researcher lowered the frequency
to 250HZ. Another dataset like the first, with EEG data from three subjects at 1000HZ. The B-CSP principle allows the researcher to
select the better EEG data in each channel using the Bhattacharyya Distance method, then extract features using the CSP algorithm,
which are into a back-propagation neural network (BPNN) for classification. The results show high precision in the data set.
Wang Tao [88] used CSP and wavelet packet analysis (WPA) to extract features respectively and used a support vector machine
(SVM) to classify and compare extracted features. The results show that the accuracy of feature extraction using the CSP algorithm
is 85.5%, and the accuracy of feature extraction using wavelet packet analysis is 92%.

5.5. Advantages and disadvantages of feature extraction methods

In recent years, researchers have done in-depth research on the feature extraction methods of EEG signals, such as adaptive au-
toregression, fast Fourier transform, wavelet transform, common spatial pattern, power spectral density, nonlinear dynamic analysis,
and so on. However, because of the non-stationary and nonlinear characteristics of EEG signals, feature extraction of EEG signals is
suitable for time-domain analysis and nonlinear methods. The traditional feature extraction method estimates its power spectrum
directly by Fourier transform. Although it is easy to implement, it has limited solving ability.
The FFT method effectively analyzes deterministic and stationary signals but has some limitations in exploring the spectrum of
abrupt movements. The principle of the AAR model is that the parametric method changes with each input at this point. However, this
method is more suitable for analyzing stationary EEG signals, and the model has limitations for motor imagery EEG, which contains
highly non-stationary signals.
The time resolution of wavelet transform at low frequency is low, but the frequency resolution is high. The time resolution is
higher at high frequency. In comparison, the frequency resolution is lower, which accords with the characteristics of low-frequency
signals changing slowly while high-frequency signals change quickly. Wavelet transform is more suitable for analyzing non-stationary
EEG signals. CSP algorithm aims to design a spatial filter to maximize the variance difference between two groups of EEG signals
after matrix filtering. This method is most suitable for EEG data of motor imagery. Different feature extraction methods have their
characteristics, and the specific advantages and disadvantages of EEG signal feature extraction are in Table 3 below.

6. Emotion classification of EEG signals

Emotions reflect human brain activity, and different emotional states have various EEG features. Emotion classification is based
on EEG features by the classifier. The commonly used methods for emotion recognition include Support Vector Machine (SVM),
Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Linear Discriminant Analysis (LDA), and Long Short-term
Memory Neural Network (LSTM).

6.1. Support vector machine

Support Vector Machines (SVM) is a binary machine learning model in statistical classification and regression analysis. SVM is a
linear classifier, and the classification object is linearly separable. Its basic model is the linear classifier with the most considerable
interval defined in the feature space. The learning strategy of SVM is interval maximization. The basic idea of SVM is to solve the

Table 3
Advantages and disadvantages of EEG signal feature extraction algorithm.

Feature extraction Advantages Disadvantages References

AAR The model is a good reflection of the state of the brain The efficiency is not high [74,75]
FFT It helps analyze stationary signals Not suitable for unsteady signals [77,78]
WT The researcher can use the signal processing in the time The calculation is too slow [82–84]
and frequency domains for feature extraction of the erratic
movement.
CSP High computational efficiency A large amount of calculation [87,88]

140
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

separated hyperplane, which can divide the training data set correctly and has the maximum geometric interval. The basic idea is
to solve the separation hyperplane, which can rightly divide the training data set and has the most significant geometric break. The
closest point to the hyperplane is called the support vector. This classification strategy aims to set up as much redundant space for
noise as possible to improve the robustness of the classifier while ensuring the correct classification of training data.
Kairui Guo [89] applied fuzzy cognitive graph (FCM) and SVM to emotion recognition. The method uses a combination of facial
expressions and EEG signals to identify emotions. The experiment was on the DEAP dataset. The experimental principle is that the
researchers cut the data into short-time data after denoising and then compress the data for dimensionality reduction. The wavelet
transform method was used to extract features and then applied to the process for identification.
S, Thejaswini [90] used SVM to conduct an experimental comparison of the DEAP dataset and SEED-IV dataset. The DEAP dataset
was by 32-channel devices Twente and Geneva, and the signal was at 500HZ. The SEED-IV dataset was from EEG signals generated
by watching 2-minute film clips from 15 subjects at 1000HZ. The researchers first lowered the frequency of the signals using filters
and MATLAB, then extracted features using discrete wavelet transform, and finally classified emotions using SVM. The results show
that the SVM method achieves good accuracy on both datasets.

6.2. Linear discriminant analysis

The linear classifier is the most commonly used in data classification, also known as linear discriminant analysis (LDA), which can
distinguish features based on Fisher distance [91]. The basic principle of LDA is to project data on low dimensions. After projection,
the projection points of each category data should be as close as possible, and the distance between category centers of data of
different categories should be as considerable as possible. The main goal of LDA is to minimize the ratio of variance within classes
and maximize the percentage of the conflict between classes. The calculation process of LDA is to calculate the inter-class matrix and
intra-class first, then calculate the eigenvalues and eigenvectors of signals, and finally construct the matrix for projection.
Masood N [92] proposed a method based on CSP and LDA to analyze emotions. Emotive EPOC records collected experimental
data. The data was into two parts. The first part of the data was from EEG signals generated by the subjects’ self-imagined images of
fear. The second part of the data was from EEG signals generated by the participants’ fear video stimulation. The researchers analyzed
the data using MATLAB, during the experiment. The researchers used the CSP algorithm during the experiment to extract features
and LDA to classify emotions. The results show an excellent classification effect.

6.3. Convolutional neural network

Many classical machine learning algorithms have been used in emotion recognition and achieved some results. However, machine
learning has disadvantages such as difficulty in feature extraction, low accuracy, and poor stability. Studies show that deep learning
is an efficient emotion recognition method and is suitable for analyzing and recognizing physiological signals. Deep learning has been
in emotion recognition due to its strong learning ability and adaptability [93].
Convolutional neural network (CNN) is a typical deep learning algorithm. Due to its unique structure, it is very suitable for sce-
narios with large data volumes. CNN neural network contains many neurons, which and presented in a three-dimensional coordinate
system [94]. Unlike the standard three-layer network structure of the input layer, hidden layer, and output layer of artificial neural
network, CNN extends the network structure based on the artificial neural network. CNN adds two network structures, the convo-
lution layer and the pooling layer, between the input layer and the full connection layer. The core network structure of CNN is the
convolution layer. The convolution layer can effectively enhance feature extraction of EEG data and reduce the influence of noise in
EEG data [95]. The main functions of the pooling layer are feature selection and information filtering. After feature extraction and
selection, the full-connection layer combines features as the input of the output layer. The output layer classifies the output using
Softmax functions. The structure of CNN in Fig. 5.
Heekyung Yang [96] proposed a multi-column CNN emotion recognition model composed of multiple recognition modules based
on convolution and pooling layers in a one-dimensional convolutional neural network. The researchers used DEAP data set for
the experimental data. The researchers preprocessed the data with reduced frequency, analyzed and sampled the EEG signals with a
bandpass filter several times, and then used them as the input of the recognition module. The recognition results will be weighted and
averaged by each module’s decisions to ensure the results’ accuracy. The model code uses Python and Pytorch libraries. Experimental
results show that the model achieves a high valence rate of 90.01% and arousal rate of 90.65%. Potency and arousal map the emotions
in Fig. 2.

Fig. 5. The structure diagram of convolutional neural network.

141
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

Yaqing Zhang [97] compared several classical emotion recognition models, including DNN, CNN, LSTM, and CNN-LSTM. The
researchers reduced the data frequency to 4–45 Hz through multiple samples and a band-pass filter. The data is into the original data
set and the shared data set. The standard data set is by feature extraction of the original data, and the two sets of data are input
into the model respectively to compare results. Researchers classified emotions based on titer and arousal, and Adam was used to
adjusting the structure and optimize parameters of the neural network. The experimental results show that the classification accuracy
of CNN and CNN-LSTM in the model is 90.12% and 94.17%, respectively, on the original dataset and 82.24% and 88.87% on the
standard dataset. The results show that CNN and CNN-LSTM have excellent performance in emotion classification.

6.4. Recurrent neural network

Recurrent neural network (RNN) is a deep learning network with a unique structure. Most neural networks consider the current
input’s influence, not the information’s impact at other times. In particular, RNN has the function of "memory," which considers inputs
at any moment. The inputs before and after are related. The network structure of RNN is relatively simple, with only three layers:
input layer, hidden layer, and output layer. RNN can work with other neural networks. In the mixed model, the researchers can use
RNN to extract the temporal characteristics of EEG signals. After feature extraction and selection, the whole connection layer of RNN
is the input of the output layer. The output layer classifies the output using the Softmax functions.
The "memory" function of RNN is in the hidden layer. RNN will remember the output of the previous moment and apply it to the
current calculation. That is, the input of the hidden layer is by the output information of the previous one and the output information
of the current input layer. Because of this, RNN is very good for processing sequential data, such as making predictions about the next
segment of a video or language. Sequential data is subsequent data related to previous data, such as continuous natural language and
long text sections in documents.
Esmeralda Contessa Djamal [98] proposed an emotional model combining WT and RNN, which can identify sadness, relaxation,
and pleasure. The dynamic model is built based on Python and TensorFlow. The researchers used wavelet transform to break down
the EEG signals into mood-related brain waves into an RNN model for recognition. The data came from EEG signals generated by
emotional audio-visua stimuli in 10 healthy adults. The researchers collected four channels of EEG data to ensure the results were
reliable. The experimental results showed that the recognition rate of sadness, relaxation, and happiness was 92%, 53%, and 97%.
The results show that the wavelet transform and RNN emotion model have excellent classification accuracy.
Wei Tao [99] proposed a cyclic convolution network (ACRNN) model combining CNN and RNN with an attention mechanism.
In the ACRNN model, CNN is mainly used to extract spatial features of EEG data, while RNN is primarily used to remove temporal
features of EEG data, both of which be emotional features. DEAP and DREAMER are experimental data. The DREAMER dataset consists
of EEG and ECG signals generated by emotional audio-visua stimulation of 23 members. Blind source separation was to reduce EEG
signal noise. The researchers built the ACRNN model according to Python and TensorFlow framework, and Adam was used to adjust
the structure and optimize neural network parameters. The Softmax function in RNN completed emotion classification. The results
show that CNN and CNN-LSTM have excellent performance in emotion classification.

6.5. Long short-term memory neural network

The traditional machine learning model has the disadvantages of complicated calculation and low efficiency [100]. Long short-
term memory neural network (LSTM) in deep learning [101] shines in emotion recognition due to its unique neural network structure.
LSTM is a particular form of the recurrent neural network [102]. Unlike RNN’s single cyclic body structure, LSTM is a neural network
combined with a "gate" mechanism [103]. The design of LSTM consists of an input layer, a circular body structure, and an output
layer, which controls the discarding or adding of information through a "gate, "thus realizing the function of forgetting or memory
[104]. LSTM unit structure is Fig. 6.
LSTM is implemented primarily through a neural layer of Sigmoid and a point-by-point multiplication operation.
Forgetting gate: The left side of Fig. 6 is the forgetting gate. ht-1 represents the state of the hidden layer at the last moment. xt
represents the current input information. The input is through the forget gate. Ct-1 represents the cell state at the last moment, and
Ct represents the cell state at the present moment. The forgetting gate determines how much of the cell state from the last moment
is to the cell state from the current moment. The forgetting gate is by the value of ft to indicate the degree of forgetting. 0 stands for
complete forgetting, and 1 stands for total memory.
( [ ] )
ft = σ Wf × ht−1 , xt + bf (4)
where σ is the Sigmoid function, Wf and bf are the weight parameters and bias items of the Sigmoid function.
Input gate: In the middle of Fig. 6 is the input gate. it represents the input state, and its value of it determines how much data is
input to the current unit. 0 indicates that the input data is none, and 1 indicates that all data is input. The tanh function is responsible
for mapping essential information to the range [−1,1]. These two parts together decide to update the status.
( [ ] )
it = σ Wi × ht−1 , xt + bi (5)
where Wi is the weight of the input gate, and bi is the offset term of the input gate.
Output gate: The output gate on the right side of Fig. 6. The output gate determines the output value and how much of the current
unit state Ct goes into the hidden state ht . ot indicates the output state and controls the output information, 0 indicates that the output
data is none, and 1 indicates that all data is output.

142
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

Fig. 6. The structure diagram of long short-term memory neural network.

The output gate determines the output value of the neural network. Firstly, the Sigmoid function is to determine the information
to be output. Tanh processes cell state Ct to obtain a weight between −1 and 1. The output is multiplied by the output information
specified by the Sigmoid function to get the final output result.
[ ]
ot = σ(Wo × ht−1 , xt + bo ) (6)

( )
ht = ot × tanh Ct (7)

where Wo is the weight of the output gate, and bo is the offset term of the output gate.
Liu [105] proposed an emotion classification model based on Bi-LSTM. Bi-LSTM combines the forward and backward information
of the input sequence based on LSTM. Bi-LSTM combines forward LSTM with backward LSTM to form Bi-LSTM. The experiment was
on a DEAP dataset. The researchers selected EEG data of 15 channels related to emotion, lowered the frequency to 128HZ, and then
filtered the data with a filter, which as the input of Bi-LSTM for recognition and classification by the Softmax layer of Bi-LSTM.
Lu [106] proposed an emotion recognition model combining a convolutional neural network (CNN) and LSTM. Experimental data
from a publicly available SEED dataset. The researchers collected the data from electrical signals generated in the brain centers of 15
healthy subjects as they watched movie clips. The dataset included positive, negative, and neutral emotional cues. In this model, the
CNN network extracts the features of 62 channel signals. In contrast, the LSTM network further integrates and extracts the features
according to the connection between the signal features of each channel. Researchers then used the Softmax function to classify the
output results of CNN-LSTM. Experimental results show that this method has a high classification rate. The suitable environment was
Ubuntu, and the emotion recognition model by TensorFlow deep learning.

6.6. Advantages and disadvantages of emotion classification methods

There are many ways to classify emotions, each of which has advantages and disadvantages. It is the key to improving the
accuracy of emotion classification to choose the classification method according to the actual situation. SVM is a traditional emotion
classification method. It is a supervised learning method with a good classification effect, but there are difficulties in classifying
large-scale data. LDA is a supervised learning classification method that relies on variance but is unsuitable for data classification
with the non-Gaussian distribution.
CNN is one of the classical algorithms of deep learning, which extracts and selects data features through the convolution layer
and pooling layer. CNN is very suitable for high-dimensional data, but the feature selection of the pooling layer may lose a lot
of valuable information. RNN is a neural network with a "memory" function, which is very suitable for sequential data. However,
gradient disappearance and explosion are in RNN. LSTM is an algorithm in deep learning and an optimized neural network of the
recurrent neural network, which is suitable for learning time series.
At present, the combination of deep learning and brain-machine interface is also one of the research hotspots. The advantages
and disadvantages of the emotion classification algorithms are in Table 4 below.

7. Existing problems and future directions

The research on emotion recognition based on EEG signals is developing rapidly. From the current situation, the research in this
area is still in the experimental stage. There is still a long time for social application, and some decisive issues need further study.
Currently available public datasets have the disadvantages of the small number of subjects, small data, and biased coverage.
Many researchers conduct experiments using their EEG data, which are difficult to analyze, compare, and unreliable. Therefore,

143
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

Table 4
Advantages and disadvantages of emotion classification methods.

EEG signal feature classifier Advantages Disadvantages References

SVM The classification idea is simple and easy to calculate Not suitable for large-scale data and [89,90]
multi-classification problems
LDA Good classification effect Data may be over-fitted [92]
CNN High dimensional data without pressure It is easy to fall into an optimal local solution [96,97]
RNN Great for processing sequential data Gradient disappearance and explosion [98,99]
LSTM Suitable for learning time series Poor parallelism and time-consuming computation [105,106]

establishing a comprehensive range, slight difference, and a large amount of public data set is conducive to developing the field of
emotion recognition.
Usually, the EEG signal is by the wet electrode method. Because the cuticle has a large impedance, issues need to apply conductive
adhesive on the scalp. However, the electrical conductivity gradually decreases with the acquisition time, and the collected EEG signal
will also become weak, which seriously affects the experimental process. Dry electrode acquisition technology is relatively simple,
but the collected signal is weak.
There is an improved dry electrode acquisition technique. The principle of this method is to compress the data as experimental
data after amplifying the collected signals and filtering them with a filter. The design is still improving. The process of EEG acquisition
is easily affected by any noise, and the existing denoising methods usually only target one kind of noise. In practical applications, the
denoising algorithm has higher accuracy requirements.
In most experiments, The researchers collected EEG signal data from full-channel EEG signals. The whole channel signal is not
conducive to experimentation and affects the accuracy of experimental results. It is unclear which channels can best reflect the
changes in emotion, and channel selection technology is a current research hotspot.
In recent years, with the rapid development of deep learning, researchers have made some achievements in applying deep un-
derstanding to emotion recognition based on EEG signals. Various mixed neural network emotion recognition models emerge in an
endless stream. The current research hotspot is a hybrid neural network model based on deep learning. By comparing the experimen-
tal results, the researchers found that the hybrid neural network model has better accuracy and stability. But the research is still in its
infancy, and different subjects and ways of inducing emotions have important implications for the results. Suppose emotion recogni-
tion technology is to society. In that case, it is necessary to establish an appropriate emotion recognition framework to eliminate or
reduce individual differences among people from all walks of life, such as age, region, personality, etc.
Generally speaking, the research on emotion recognition based on EEG signals is still in its infancy, and there is a broad space for
development. Emotion affects every aspect of human life. This research has excellent application prospects and research value. At
present, how to improve the efficiency of EEG signal acquisition and emotion recognition rate is still a complex problem to be solved.

8. Conclusion

As a branch of emotion computing, emotion recognition is the foundation and core technology of human-computer emotion
interaction. Emotion affects all aspects of human life, so the study of emotion recognition is of great application value. In recent
years, brain-computer interface technology has been making continuous progress with the development of artificial intelligence.
Emotion recognition based on EEG has a broad application prospect and has received extensive attention. This paper introduces the
research progress of emotion recognition based on EEG signals. The acquisition and preprocessing of EEG signals affect the accuracy
of emotion classification, and the choice of classification method also determines the result of emotion classification.
At present, deep learning has been successfully applied in emotion recognition. Researchers combine deep learning with the field
of brain-computer interface and propose various emotion recognition models based on neural networks. Researchers combine deep
learning with the area of brain-machine interface and submit different emotion recognition models based on hybrid neural networks.
Combining emotion recognition and deep learning based on EEG signals has become a new research direction.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to
influence the work reported in this paper.

Acknowledgments

This work is supported by Project of the Chinese Society of Academic Degrees under grant B-2017Y0002-170, and Key Project of
Shaanxi Science and Technology under grant 2016GY-040.

References

[1] R.A. Calvo, S. D’Mello, Affect detection: an interdisciplinary review of models, methods, and their applications, IEEE Trans. Affect. Comput. 1 (1) (2010) 18–37.
[2] R.W. Picard, Affective computing: challenges, Int. J. Hum. - Comput. Stud. 59 (1–2) (2003) 55–64.

144
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

[3] J. Chen, Classification of brain electrical movement imagination based on deep convolution network, China Med. Dev. (2019).
[4] S.M. Alarcão, M.J. Fonseca, Emotions recognition using EEG signals: a survey, IEEE Trans. Affect. Comput. (1949) 1-1.
[5] C. Shahnaz, S.B. Masud, S. Hasan, Emotion recognition based on wavelet analysis of Empirical Mode Decomposed EEG signals responsive to music
videos//TENCON 2016 - 2016 IEEE Region 10 Conference, IEEE, 2016.
[6] X. Yang, J. Liu, Z. Chen, et al., Semi-supervised Learning of Dialogue Acts Using Sentence Similarity Based on Word Embeddings, IEEE, 2015.
[7] I.A. Adeyanju, E.O. Omidiora, O.F. Oyedokun, Performance Evaluation of Different Support Vector Machine Kernels for Face Emotion Recognition//Sai Intel-
ligent Systems Conference, IEEE, 2015.
[8] Y. Cao, H. Tang, Y. Ma, et al. The Development and Modern Application of the EEG Signal Acquisition System. New Technology & New Process, 2015.
[9] D.G. Kim, K.S. Hong, K.W. Chung, Implementation of portable multi-channel EEG and head motion signal acquisition system//International Conference on
Computing & Networking Technology, IEEE, 2013.
[10] R.W. Picard, E. Vyzas, J. Healey, Toward machine emotional intelligence: analysis of affective physiological state, IEEE Trans. Pattern Anal. Mach. Intell. 23
(10) (2001) 1175–1191.
[11] S. Koelstra, DEAP: a database for emotion analysis; using physiological signals, IEEE Transactions on Affective Computing, 2012.
[12] P.V. Mierlo, Y. Hller, N.K. Focke, et al., Network perspectives on epilepsy using EEG/MEG source connectivity, Front. Neurol. (2019) 10.
[13] S. Weisdorf, J. Duun-Henriksen, M.J. Kjeldse, et al., Ultra-long-term subcutaneous home monitoring of epilepsy—490days of EEG from nine patients, Epilepsia
(2019) 60.
[14] L.X. Zhang, T. Liu, F.Q. Pan, et al., Analysis of effects of driver factors on road traffic accident indexes, China Saf. Sci. J. (2014).
[15] C. Pêcher, C. Lemercier, J.M. Cellier, Emotions drive attention: effects on driver’s behaviour, Saf. Sci. 47 (9) (2009) 1254–1259.
[16] E. Ambron, R.I. Rumiati, F. Foroni, Do emotions or gender drive our actions? A study of motor distractibility, Cogn Neurosci 7 (1–4) (2016) 160–169.
[17] A.S. Kanagalakshmi, M. Dinesh, M.S. Raju, et al. Analysis of road accidents national highway-5.
[18] J. Cheng, Y. Zhang, Prediction of driving behavior under different emotions, J. Transp. Eng. Inf. (2019).
[19] M.E. Zhong, H.C. Hong, Z.Q. Yuan, Experiment Research on Influence of Angry Emotion for Driving Behaviors, Journal of Chongqing University of Technology
(Natural Science, 2011.
[20] L. Li, W.T. Wang, Y.H. Shi, Influences of emotion on drivers’attention based on eye movement experiment, J. Xi’an Univ. Sci. Technol. 4l (1) (2021) 70–74.
[21] K. Schaaff, EEG Based Emotion recognition, Diplomarbeit am Institute for Algorithms and Kognitive System, University at Karlsruhe, 2008.
[22] C.Y. Chang, J.S. Tsai, C.J. Wang, P.C. Chung, Emotion recognition with consideration of facialexpression and physiological signals. in, in: Proceeding of IEEE
Symposium on Computational Intelligence in Bio infomatics and computational biology, 2009, pp. 278–283.
[23] P.C. Petrantonakis, L.J. Hadjileontiadis, Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis, IEEE Trans.
Affect. Comput. 1 (2) (2011) 81–97.
[24] D. Huang, S. Zhang, Y. Zhang, EEG-based emotion recognition using empirical wavelet transform//2017, 4th International Conference on Systems and Infor-
matics (ICSAI), IEEE, 2018.
[25] G. Placidi, D. Avola, A. Petracca, et al., Basis for the implementation of an EEG-based single-trial binary brain computer interface through the disgust produced
by remembering unpleasant odors, Neurocomputing 160 (21) (2015) 308–318 jul..
[26] P. Ekman, R.W. Levenson, W. Friesen, Autonomic nervous system activity distinguishes among emotions, Science 221 (4616) (1983) 1208–1210.
[27] P.J. Lang, M.M. Bradley, B.N. Cuthbert. International Affective Picture System (IAPS): affective ratings of pictures and instruction manual (Rep. No. A-8). 2008.
[28] M. Bradley, P. Lang, The International Affective Digitized Sounds: Affective Ratings of Sounds and Instruction Manual, University of Florida, 2007.
[29] Y.J. Li, EEG Signal Analysis Method and its Application, Science Press, Beijing, 2009.
[30] A. Konar, A. Chakraborty. Emotion recognition (A Pattern Analysis Approach) || bodily expression for automatic affect recognition. 2015,
10.1002/9781118910566:343-377.
[31] E. Lalor, S.P. Kelly, C. Finucane, et al., Brain computer interface based on the steady-state VEP for immersive gaming control, in: Proceedings of the 2nd
International Brain-Computer Interface Workshop and Training Course, 49, 2004, pp. 63–64.
[32] R.P. Hazimah, Z. Hasan, F.A. Jafar, et al. A comparison of SSDXCOR and CC matching techniques for vision inspection system. Adv. Sci. Lett., 2017.
[33] S.N. Deraman, N.V. David, Effectiveness of isochronic tones on gamma-domain brainwave stimulation, Iran. J. Pharm. Sci. 15 (0) (2020) 5.
[34] A. Samavat, E. Khalili, B. Ayati, et al., Deep learning model with adaptive regularization for EEG-based emotion recognition using temporal and frequency
features, IEEE Access 10 (2022) 24520–24527.
[35] K. Chen, M. Chen, L. Cheng, et al., A 124-dB dynamic range sigma-delta modulator applied to non-invasive EEG acquisition using chopper-modulated input-s-
caling-down technique, Sci. China Inf. Sci. 65 (4) (2022) 1–15.
[36] A. Yazdani, J. Lee, T. Ebrahimi, Implicit emotional tagging of multimedia using EEG signal sandbrain computer interface ll, in: Proceedings of the first SIGMM
Workshop Onsocial Media.Beijing,China, ACM, 2009, pp. 81–88.
[37] P. Zhong, D. Wang, C. Miao, EEG-based emotion recognition using regularized graph neural networks, IEEE Trans. Affect. Comput. (99) (2020) 1-1.
[38] D.M. Tucker, M. Liotti, G.F. Potts, et al., Spatiotemporal analysis of brain electrical fields, Hum. Brain Mapp. 1 (2) (1994).
[39] A. Craig, Y. Tran, N. Wijesuriya, et al., Regional brain wave activity changes associated with fatigue, Psychophysiology 49 (4) (2012) 574–582.
[40] T. Kazutaka, S. Maryam, R.D. Penn, et al., Propagating waves in human motor cortex, Front. Hum. Neurosci. 5 (2011) 40.
[41] S. Halder, D. Agorastosa, R. Veita, et al., Neural mechanisms of brain-computer interface control, Neuroimage 55 (4) (2011) 1779–1790.
[42] X. Jia, A. Kohn, et al., Gamma Rhythms in the Brain, PLoS Biol. (2011).
[43] K.J. Panoulas, L.J. Hadjileontiadis, S.M. Panas. Brain-Computer Interface (BCI): types, processing perspectives and applications. Springer Berlin Heidelberg.
[44] S. Qiu, W. Yi, H. Qi, Event-related beta EEG Changes during active, passive movement and functional electrical stimulation of the lower limb, IEEE Trans.
Neural Syst. Rehabil. Eng. 24 (2) (2016) 1-1.
[45] G.K. Verma, U.S. Tiwary, Affect representation and recognition in 3d continuous valence - arousal -dominance space, Multimed. Tools Appl. 76 (2) (2017)
2159–2183.
[46] B. Mojtaba, B. Richard, G.F. Carolina, et al., Dynamic modulation of theta–gamma coupling during rapid eye movement sleep, Sleep (12) (2019) 1–11.
[47] M. Li, B.L. Lu, Emotion classification based on gamma-band EEG, in: Conference: International Conference of the IEEE Engineering in Medicine & Biology
Society IEEE Engineering in Medicine & Biology Society Conference, IEEE, 2009, pp. 1223–1226.
[48] A.I. Triggiani, V. Bevilacqua, A. Brunetti, et al., Classification of healthy subjects and Alzheimer’s disease patients with dementia from cortical sources of resting
state EEG rhythms: a study using artificial neural networks, Front. Neurosci. 10 (2017) 604.
[49] P. Zanatta, P. Tellaroli, M. Carron, et al. Isolated forearm technique: a meta-analysis of connected consciousness during different general anaesthesia regi-
mens.2013.
[50] S. Hagihira, Brain mechanisms during course of anesthesia: what we know from EEG changes during induction and recovery, Front. Syst. Neurosci. 11 (2017)
39.
[51] K. Han, H. Wen, Y. Zhang, et al., Deep predictive coding network with local recurrent processing for object recognition, Adv. Neural Inf. Process. Syst. (2018)
31.
[52] A. Clark, Whatever next? Predictive brains, situated agents, and the future of cognitive science, Behav. Brain Sci. 36 (3) (2013) 181–204.
[53] T.P. Jung, S. Makeig, M. Westerfield, et al. Removal of eye activity artifacts from visual event-related potentials in normal and clinical subjects. 2000.
[54] P.A. Babu, K. Prasad, Removal of ocular artifacts from EEG signals using adaptive threshold PCA and wavelet transforms, International Conference on Commu-
nication Systems & Network Technologies, IEEE, 2011.
[55] S.S. Poorna, V.V. Arsha, P.T.A. Aparna, et al. Drowsiness detection for safe driving using PCA EEG signals. 2018.
[56] E. Javed, I. Faye, A.S. Malik, et al., Removal of BCG artefact from concurrent fMRI-EEG recordings based on EMD and PCA, J. Neurosci. Methods (2017) 150.
[57] M.A. Rahman, M.F. Hossain, M. Hossain, et al., Employing PCA and t -statistical approach for feature extraction and classification of emotion from multichannel
EEG signal, Egyptian Informatics Journal 21 (1) (2020) 23–35.

145
C. Yu and M. Wang Cognitive Robotics 2 (2022) 132–146

[58] Z. Li, et al., Source separation method of machine faults based on post-nonlinear blind source separation. 2008, 7th World Congress on Intelligent Control and
Automation, IEEE, 2008.
[59] A. Hyvärinen, Survey on independent component analysis, Neural Comput. Surv. (1999).
[60] Z. Liu, Z. Miao, L. Wan, Speech blind signal separation with FastICA and Markov Chain combination//2014, in: 12th International Conference on Signal
Processing, IEEE, 2014, pp. 541–544. ICSP.
[61] D.J.R. Bouveresse, D.N. Rutledge. Independent components analysis: theory and applications. 2016.
[62] P.S. Prasad, Independent Component Analysis, Cambridge University Press, 2001.
[63] S. Vorobyov, A. Cichocki, Blind noise reduction for multisensory signals using ICA and subspace filtering, with application to EEG analysis, Biol. Cybern. 86
(2002) 293–303.
[64] A. Vergult, W.D. Clercq, A. Palmini, et al., Improving the Interpretation of Ictal Scalp EEG: BSS–CCA Algorithm for Muscle Artifact Removal, Epilepsia 48 (5)
(2007) 950–958.
[65] J. Gao, L. Pan, Y. Yong, et al., Real-time removal of ocular artifacts from EEG based on independent component analysis and manifold learning, Neural Comput.
Appl. 19 (8) (2010) 1217–1226.
[66] M.J. Cannon, D.B. Percival, D.C. Caccia, et al., Evaluating scaled windowed variance methods for estimating the Hurst coefficient of time series, Physica A 241
(3–4) (1997) 606.
[67] C. Kaur, P. Singh, S. Sahni, EEG artifact removal system for depression using a hybrid denoising approach, Basic Clin. Neurosci. 12 (4) (2021) 465–476.
[68] R.J. Smith, D.K. Hu, D.W. Shrey, et al., Computational characteristics of interictal EEG as objective markers of epileptic spasms, Epilepsy Res. (11) (2021)
106704.
[69] J. Sijbers, J.V. Audekerke, M. Verhoye, et al., Reduction of ECG and gradient related artifacts in simultaneously recorded human EEG/MRI data, Magn. Reson.
Imaging 18 (7) (2000).
[70] S. Tong, A. Bezerianos, J. Paul, et al., Removal of ECG interference from the EEG recordings in small animals using independent component analysis, J. Neurosci.
Methods 108 (1) (2001) 11–17.
[71] C. Dai, J. Wang, J. Xie, et al., Removal of ECG artifacts from EEG using an effective recursive least square notch filter, IEEE Access (99) (2019) PP1-1.
[72] Z. Jiao, X. Gao, Y. Wang, et al., Deep convolutional neural networks for mental load classification based on EEG data, Pattern Recognit. 76 (2018) 582–595.
[73] S. Siuly, Y. Li, Designing a robust feature extraction method based on optimum allocation and principal component analysis for epileptic EEG signal classification,
Compute Methods Programs Biomed. 119 (1) (2015) 29–42.
[74] H. Li, J. Wang, B. Deng, et al., Feature extraction method based on AAR model and accumulated band power, J. Tianjin Univ. 46 (9) (2013) 784–790.
[75] Q. Zhang. EEG-based classification of brain activity for brightness stimuli//Springer Berlin Heidelberg. Springer Berlin Heidelberg, 2009.
[76] A.S. Al-Fahoum, A.A. Al-Fraihat, Methods of EEG signal features extraction using linear analysis in frequency and time-frequency domains, ISRN Neurosci.
(2014) 730218.
[77] M.K. Delimayanti, B. Purnama, N.G. Nguyen, et al., Classification of brainwaves for sleep stages by high-dimensional FFT features from EEG signals, Appl. Sci.
10 (5) (2020).
[78] E.C. Djamal, W.I. Furi, F. Nugraha, E.C. Djamal, W.I. Furi, F. Nugraha, in: Detection of EEG signal post-stroke using FFT and convolutional neural network,
Proceeding of the Electrical Engineering Computer Science and Informatics, 2019, p. 6.
[79] J. Yao, Z. Luo, S.O. Automation, et al., Application of dual-tree complex wavelet transform in EEG denoising, J. Huazhong Univ, Sci. Technol. (2015).
[80] V. Bono, W. Jamal, S. Das, et al., Artifact reduction in multichannel pervasive EEG using hybrid WPT-ICA and WPT-EMD signal decomposition techniques,
IEEE International Conference on Acoustics, IEEE, 2014.
[81] Q. Zhao, B. Hu, Y. Shi, et al., Automatic identification and removal of ocular artifacts in EEG—improved adaptive predictor filtering for portable applications,
IEEE Trans. Nanobiosci. 13 (2) (2014) 109–117.
[82] Z. Ma, Feature extraction and classification of motor imagine EEG signals based on sample entropy, Shandong Ind. Technol. (2016) 289–291.
[83] X. Niu, Q. Ye, Y. Zhou, et al., EEG signal recognition based on genetic algorithm feature selection in autoregressive model, Comput. Eng. 42 (3) (2016) 283–294.
[84] Y. Hu, L. Wang, W. Fu, EEG feature extraction of motor imagery based on WT and STFT//2018, IEEE International Conference on Information and Automation
(ICIA), IEEE, 2018.
[85] K.P. Thomas, C. Guan, C.T. Lau, et al., A new discriminative common spatial pattern method for motor imagery brain–computer interfaces, IEEE Trans. Biomed.
Eng. 56 (11) (2009) 2730–2733.
[86] F. Lotte, C. Guan, Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithm, IEEE Trans. Biomed. Eng. 58 (2) (2010)
355–362.
[87] Z. Tang, C. Li, J. Wu, et al., Classification of EEG-based single-trial motor imagery tasks using a B-CSP method for BCI, Front. Inf. Technol. Electron. Eng. 20
(8) (2019) 1087–1098.
[88] T. Wang, L.Y. Wu, Y.P. Li, et al., Learning Advanced brain computer interface technology: comparing CSP algorithm and WPA algorithm for EEG feature
extraction, Int. J. Technol. Hum. Interact. 15 (3) (2019) 14–27.
[89] K. Guo, H. Yu, R. Chai, et al., A hybrid physiological approach of emotional reaction detection using combined FCM and SVM classifier//2019 41st, Annual
International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), IEEE, 2019.
[90] D.K.M. Kumar, J.L. Nataraj. Analysis of EEG based emotion detection of DEAP and SEED-IV databases using SVM. 2019.
[91] R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, Wiley, 2004.
[92] N. Masood, H. Farooq, Investigating EEG patterns for dual-stimuli induced human fear emotional state, Sensors 19 (3) (2019).
[93] Y. LeCun, B. Boser, J.S. Denker, et al., Backpropagation applied to handwritten zip code recognition, Neural Comput. 1 (4) (1989) 541–551.
[94] S. Tripathi, S. Acharya, R.D. Sharma, et al. Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. 2017.
[95] J.P. Li, Z.X. Zhang, H.G. He, Hierarchical convolutional neural networks for EEG-based emotion recognition, Cognit. Comput. 10 (2) (2018) 368–380.
[96] H. Yang, J. Han, K. Min, A multi-column CNN model for emotion recognition from EEG signals, Sensors 19 (21) (2019) 4736.
[97] Y. Zhang, J. Chen, J.H. Tan, et al., An investigation of deep learning models for EEG-based emotion recognition, Front. Neurosci. (2020) 14.
[98] E.C. Djamal, R.D. Putra, Brain-computer interface of focus and motor imagery using wavelet and recurrent neural networks, Telkomnika 18 (5) (2020)
2748–2756.
[99] W. Tao, C. Li, R. Song, et al., EEG-based emotion recognition via channel-wise attention and self attention, IEEE Trans. Affect. Comput. (99) (2020) PP.
[100] Z. Yin, M. Zhao, Y. Wang, et al., Recognition of emotions using multimodal physiological signals and an ensemble deep learning model, Comput. Methods
Programs Biomed. 140 (Complete) (2017) 93–110.
[101] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput. 9 (8) (1997) 1735–1780.
[102] X. Han, N. Kheir, D. Balzarotti. PhishEye: live monitoring of sandboxed phishing kits. 2016.
[103] L.I. Yang, H. Dong, Text sentiment analysis based on feature fusion of convolution neural network and bidirectional long short-term memory network, J.
Comput. Appl. (2018).
[104] C.H. Sun, B. Hu, Y.X. Zou, A BP-LSTM trend forecast model for stock index, J. Sichuan Univ. 57 (2020) 27.
[105] S. Liu, L. Wang, X. Ding, Emotional EEG recognition based on Bi-LSTM, J. Shandong Univ. 50 (4) (2020) 35–39.
[106] G. Lu, W. Cong, J. Wei, EEG based emotion recognition using CNN and LSTM, J. Nanjing Univ. Posts Telecommun. 41 (1) (2021) 58–64.

146