See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/353702181

Deep Learning Applications in Medical Image Analysis

Chapter · August 2021

DOI: 10.1002/9781119711278.ch11

CITATIONS READS

20 1,761

3 authors:

Tushar Patel Ananya Singha

Leibniz Institute on Aging – Fritz Lipmann Institute Indian Institute of Science Education and Research Bho…
5 PUBLICATIONS 26 CITATIONS 1 PUBLICATION 20 CITATIONS

SEE PROFILE SEE PROFILE

Rini Thakur
Indian Institute of Science Education and Research Bho…
10 PUBLICATIONS 419 CITATIONS

SEE PROFILE

All content following this page was uploaded by Rini Thakur on 18 January 2022.

The user has requested enhancement of the downloaded file.

 11
Deep Learning Applications in
Medical Image Analysis
Ananya Singha1, Rini Smita Thakur2 and Tushar Patel3*

Department of Engineering Sciences, Indian Institute of Science Education and

1

Research, Bhopal, India

2
Department of Electronics and Communication Engineering, Maulana Azad
National Institute of Technology, Bhopal, India
3
Department of Biological Sciences, Indian Institute of Science Education and
Research, Bhopal, India

Abstract
Deep learning is slowly taking over the medical image analysis field with advance-
ments in imaging tools, and growing demand for fast, accurate, and automated image
analysis. In this chapter we have extensively reviewed the field from inception to its
current state-of-the-art techniques. We begin by introducing imaging advancements
over the decades, followed by general pre-analysis processes. Further, we describe the
different kinds of deep learning models used for image analysis, after broadly clas-
sifying them into supervised and unsupervised learning. Here the chapter touches
upon the architecture and complex functionality of each layer of such a deep neu-
ral network. CNN being one of the most popular tools for image analysis is then
reviewed in-depth. We have discussed its architecture and learning along with the
mathematical framework behind it, followed by real-world applications. The second
half of this chapter is devoted to the biological applications of deep learning. We start
by reviewing advancements from sub-cellular to cellular to the tissue level, ending
with an extensive review at the organ level. We have included various deep learning
tools and techniques that people have attempted to better understand the morphol-
ogy and physiology of the human body, and its related diseases. Finally, the chapter
ends with current field-specific challenges and the ways forward to eliminate them.
Keywords: Deep learning, medical image analysis, convolutional neural
networks (CNNs), image classification, image segmentation, medical diagnosis

*Corresponding author: [email protected]; [email protected]

Sujata Dash, Subhendu Kumar Pani, S. Balamurugan and Ajith Abraham (eds.) Biomedical Data
Mining for Information Retrieval: Methodologies, Techniques and Applications, (293–350) © 2021
Scrivener Publishing LLC

293
294 Biomedical Data Mining for Information Retrieval

11.1 Introduction
Image analysis is a complex process that a human brain performs typi-
cally, at least a million times a day. The brain is a versatile machine but has
limits of speed, accuracy, and exhaustion. These natural machines’ limita-
tions compelled humans to develop tools like artificial intelligence (AI) in
the last century. Ever since this field was born, it has seen a multitude of
advancements due to ever-improving computational technology and high-
end storage with fast transmission capacity over distances. Following the
quest for training machines in an intelligent manner to let them learn from
data, introduced us to machine learning. Powered by self-evolving design,
machine learning finds excellent applications in solving many practical
problems for humans like remote sensing, agriculture, handwriting and
face recognition, disease diagnosis, traffic prediction, and many more.
With the increasing complexity of issues that we are dealing with, machine
learning is now further improved into deep learning. It is a more advanced
version of the earlier, involving more layers of complex algorithms.
Furthermore, this finds application in a yet more complex field of natural
architecture, the medicine. Here, deep learning deals with analyzing biolog-
ical systems’ images ranging from subcellular structures to irregular-sized
tumors using neural networks. The various image pre-processing steps like
image classification, registration, segmentation, denoising, decomposition,
are required to extract relevant information from the medical images. We
are going to discuss all these in great detail across this chapter.
This chapter is about deep learning and its recent advancements to aid
medical image analysis. We will discuss the variations that have evolved
across different techniques of deep learning according to challenges in spe-
cific fields, and emphasize one such extensively used tool, Convolutional
Neural Network (CNN), in medical image analysis. We would talk about
the architecture of CNN and its training and how it functions. We would
shed some light on the ways it is used for image denoising and classifi-
cation concerning medical image analysis. In the next section, we would
report various advancements in the medical image analysis field using deep
learning applications, in a hierarchical biological classification manner. We
will start by targeting small structures like organelles to bigger ones like
tissues and organs, providing a comprehensive overview. We have specially
designed this content for medical practitioners to provide them an exclusive
view of all significant techniques that have evolved in the field. This includes
a descriptive and tabular mention of level-wise novel techniques that aid in
disease recognition and treatment in earlier stages. We will include some
 Deep Learning in Medical Image Analysis 295

resources of various open-access medical image repositories available to

researchers for analysis of medical image modalities such as CT, MRI, ultra-
sound, X-rays. We will end this chapter with current challenges in the field
and possible ways to overcome them. It follows a short discussion highlight-
ing the importance of medical practitioners proving that these techniques
are not a replacement for their jobs; instead, they are an excellent aid for
them in research and diagnosis of diseases with ease and accuracy.

11.1.1 Medical Imaging

Medical Imaging is a technique of capturing morphology and physiology of
different parts of the human body, through images and graphical data com-
pilation for proper clinical analysis and medical intervention. The electro-
magnetic/acoustic waves are streamed through the human body to capture
a visual representation of anatomy. Medical imaging also includes mea-
surement and recording techniques that do not output images but produce
data in a graphical representation like Electrocardiogram. The field traces
back its history from the 19th century, with the introduction of X-rays by
Wilhelm C. Röntgen in 1895. The technique’s basic principle was the vari-
able penetrance of X-rays through human flesh and bone [1]. Since then,
many breakthroughs in the field are made, which are broadly classified into
Morphological/Anatomical and Molecular imaging techniques. The former
include ultrasound, computed tomography (CT), and magnetic resonance
imaging (MRI), which mostly observes for structural changes. The latter
includes techniques such as optical imaging, positron-emission tomography
(PET), and single-photon emission tomography (SPET), which scans for
more molecular details of the given biological system. Both types of tech-
niques have their advantages and hence are now used together in coordina-
tion to allow high structured resolution images like PET-CT or PET-MRI [2].
The basic underlying principle of tomography-based techniques like
CT, MRI, PET, SPECT is to image any biological object in thin slices and
then superimpose each fragment to form a complete image. Unlike them,
ultrasound uses high-frequency sonic waves to propagate inside biologi-
cal systems, and returned echoes are analyzed for image formation. While
on the other hand, optical imaging is a highly precise technique that uses
the absorption and emission properties of various bio-active molecules
such as green fluorescent protein (GFPs) for imaging purposes [3]. The
last decade has seen some promising advancements in existing functional
MRI (fMRI), which have improved spatial and temporal resolution limits
by many folds. The analysis of the images obtained by fMRI is used for 3D
visualization and anatomical studies [4]. Now, with the introduction of the
296 Biomedical Data Mining for Information Retrieval

time element in 3D; 4D imaging is gaining its popularity and, when com-
bined with methods CT, MRI, PET, and SPECT, it can be used in studying
fluid dynamics of internal human organs and generate structured mesh
models of them [5, 6]. In Table 11.1, advancements in the field of medical
imaging are given [1, 2, 8–11].
As shown above, the advancements in medical imaging modalities over
time led to significant data acquisition and an increased load on medi-
cal experts to analyze them, which made them prone to more errors. This
makes them prone to more errors. This thus led to the era of AI (Artificial
Intelligence), for relatively more straightforward tasks like ‘classification’ as
back as in 1995 [7]. Later machine learning, and then more advanced deep
learning tools emerged for denoising and other various pre-­processing
steps before actual image analysis. Let’s first find our way to understand
artificial intelligence and deep learning in some detail.

11.1.2 Artificial Intelligence and Deep Learning

McCulloch and Pitts in 1943 developed the first artificial neuron, with which
gradually came the realm of AI [12]. Artificial Intelligence (AI) alludes to
the invigoration of human intellectuality in machines guided to think like
humans. Also, it mimics their actions for executing tasks from the easy to
the even more complicated ones. In 1958 the first artificial neural network
was built by psychologist Frank Rosenblatt, called Perceptron [13]. It was
modeled to visualize and recognize objects cognitively. In crux, an artifi-
cial neural network is nothing but layers of connected neurons with inputs
and outputs linked to each such layer. They operate by creating connec-
tions between different elements analogous to a single neuron in the human
brain. Unlike neurons in the brain, these are created and simulated by the
computer. Neurons take inputs through a weighted system, then process
the data and send it as input to another neuron. Several neurons are tightly
coordinated into layers, and generally, two or three hidden layers are sand-
wiched in between ultimate input and output ones. This design helps it to
improve its performance and predict the results with remarkable accuracy.
The artificial neural network (ANN) is trained by different learning
methods with the help of application-oriented training datasets. Broadly,
there are two methods for training an ANN, depending upon the prob-
lem it would solve. One way is to feed the ANN with vast datasets having
some pattern or resemblance, and then it is allowed to recognize it. This
method is generally used to analyze experimental data. The other method
is to give rewards during the training period, positive for correct outputs,
and negative for the wrong output, which ends up giving the decisive
Table 11.1 Advancements in tools of medical imaging.
Imaging First used Spatial Major fields of Benefits # effects
modality (tentative) resolution Frontier technique application (if any)
X-Ray 1895 by Wilhelm C. 1 mm X-Rays (From Chiropractic/Dental Non-invasive,
Radiography Röntgen cathode) (0.1–10 Examinations, quick, painless
n.m.) + X-ray Fractures #Ionizing radiation
detector Projection, (possibility of
Mammography, cancer)
Arthrography, Bone
Densitometry
UltrasonoGraphy Late 1940s by 1 mm Pressure waves (20 Fetal development, Real-time
(Ultrasound) Dr George kHz) + Echo Solid organs information,
Ludwig detector imaging like Ability to
pancreas, gall measure
bladder etc., gradient flow,
ECG for heart High resolution
functioning
(Continued)
Deep Learning in Medical Image Analysis
297
Table 11.1 Advancements in tools of medical imaging. (Continued)
298

Imaging First used Spatial Major fields of Benefits # effects

modality (tentative) resolution Frontier technique application (if any)
SPECT Scan Around 1960–70s 12–15 mm3 Gamma decay (Radio nuclei Imaging) Highly accurate
(Single by David Kuhl radioisotope Cancer diagnosis, and specific
photon-based) and Roy Edward + 360° Photon Diagnosis of information,
(Pioneers capturing camera Cardiovascular/ Provides a
experimentalists) Alzheimer/ complete image
Parkinson/ of target site,
PET Scan Around 1972 by 4–8 mm3 Positron decay Dementia diseases, Differentiate
(Positron Gordon Brownell radioisotope + Neurological between good
based) Simultaneous disorders, and bad tissue
180° detection Evaluating #Ionizing radiation
circuitry the potential (possibility of
effectiveness of cancer)
therapy
CT Scan 1972 by Sir Godfrey 0.5 mm X-Rays (From 2D/3D/4D Body High spatial
(Computed Hounsfield (Along cathode) (0.1- examination, resolution
tomography) x,y,z-axis) 10 n.m.) + Diagnosing disease, volumetric
Computer and trauma, and imaging
Biomedical Data Mining for Information Retrieval

detector abnormality #Ionizing radiation

(possibility of
cancer)
(Continued)
Table 11.1 Advancements in tools of medical imaging. (Continued)
Imaging First used Spatial Major fields of Benefits # effects
modality (tentative) resolution Frontier technique application (if any)
MRI Scan Early 1980s by 1–2 mm Gradient Magnetic Human body Without ionizing
(Magnetic Peter Mansfield (Along field and examination, radiation,
resonance and Paul C. x,y,z-axis) radio fields + Unhealthy tissue High spatial
imaging) Lauterbur Computational retrieval, Viewing resolution,
Equipment collateral veins and Fluid dynamic
cranial, Surgery analysis, Good
planning Soft-tissue
contrast
Fluorescent Early 1900s by Carl Approx. Phosphorescence/ Steady-state Extremely high
microscopy Zeiss and Carl 0.0002, Bioluminescence organization accuracy and
Or Optical Reichert 0.0006 mm Molecules of a cell and resolution,
microscopy (Lateral, (Probes) + its dynamics; highly specific,
axial) Detection Structure, turnover, Enhances live-
sensors mobility, and cell microscopy
function of cellular
components
Deep Learning in Medical Image Analysis
299
300 Biomedical Data Mining for Information Retrieval

approach to output with maximum reward. This method is widely used for
cognitive research and problem-solving applications. The parallel network
of ANN can be built with multiple processors, providing it with a high-
speed advantage at a little development cost. ANN is extensively applied
for image recognition and speech recognition.
Deep Learning is a highly rising domain in the Artificial Intelligence (AI)
world. Networks are capable of learning in an unsupervised manner from
unstructured or unlabeled data. It is also known as deep neural networks or
deep AI, occasionally. Deep Learning or deep AI is a humongous collection of
such artificial neural networks (ANNs). With ANN’s potential, we can auto-
matically extract significant low-level features and amalgamate them into
great defining features. These advances of ANN or deep learning have opened
up applications in the domain of healthcare and medicine. The intertwining
of the pre-existing medical imaging techniques with Deep Learning has given
some remarkable results that we are going to discuss across this chapter.
Deep Learning is broadly categorized into supervised and unsupervised
learning, based on the kind of data used for training. They have different
structures and working principles. The convolution neural network (CNN)
is a complex feed-forward network under supervised deep learning that is
extensively applied for image recognition and analysis [14]. Let us start by
first understanding processing in medical images.

11.1.3 Processing in Medical Images

The accurate image analysis of biomedical images involves various image
processing operations like image segmentation, classification, enhance-
ment, registration. These operations are used for quantitative measure-
ments as well as an abstract interpretation of biomedical images.
Medical image segmentation is the technique of automatic or semi-
automatic identification of pixels associated with organs or lesions from
the background of medical images. It breaks an image into different regions
with homogenous properties such as gray level, contrast, brightness, color,
texture. It helps in the study of anatomical structure, the region of interest
identification i.e., tumor or lesion location, radiation dose calculation for
cancer treatment and, tissue volume measurement [15]. Deep learning has
integrated algorithms of edge detection, region-based segmentation, tex-
ture analysis, histogram equalization into neural networks for automatic
segmentation with increased accuracy. There are popular neural networks
involving deep learning techniques for image segmentation. This includes
convolutional neural networks, convolutional residual networks, and
recurrent neural networks [16]. Figure 11.1 depicts a segmented image of
 Deep Learning in Medical Image Analysis 301

Figure 11.1 Image depicting prostate cancer cell segmentation (Source—Alan Partion
John Hopkins University).

a prostate cancer cell. This image contains two different spatially oriented
cells, but only one of them is distinguishable. The ultimate motive is either
to detect or to segment, the evident cell.
Image Classification is the process of assigning image pixels into different
categories or classes according to application. The input images are subdi-
vided into classes based on features. The main difference between machine
learning (ML) and deep learning (DL) algorithms is the feature extraction
step. The traditional machine learning approach requires hand-crafted
features that are time-consuming, difficult, and require domain expertise,
whereas deep learning automatically extracts features by algorithms. Image
classification assessment metrics are specificity, sensitivity, and accuracy
[17]. Some examples of classification in medical imaging are classification
of a skin lesion for malignancy using Alex-Net CNN [18], classification of
hematoxylin and eosin-stained (H&E) breast biopsy images using CNNs
[19]. Some others can be to classify optical coherence tomography images
into age-related macular degeneration and diabetic macular edema. This
uses the Inception V3 network based on transfer learning [20], multi-label
chest X-ray classification using ResNet architecture [21].
Image Enhancement is generally performed to improve the overall visual
quality of medical images for an accurate diagnosis. MR images are cor-
rupted by rician noise, salt and pepper noise, Gaussian noise, and speckle
noise. Similarly, CT images are mostly corrupted by Gaussian noise, ran-
dom noise, electronic noise, statistical noise due to the statistical uncer-
tainty of physical measurements involved. This includes radiation dose,
software, and ­hardware-based artifacts, patient motion [22]. Ultrasound
images are degraded by the presence of ‘Speckle noise’, a signal-dependent
multiplicative disturbance [23]. It reduces the resolving power of image
and contrast that affects diagnostic value. PET images exhibit a higher
noise level as compared to MRI and CT scans, as it is corrupted by a mix-
ture of Gaussian and Poisson noise [24]. The image denoising techniques
302 Biomedical Data Mining for Information Retrieval

are broadly classified as spatial transform methods, morphological based

methods, fuzzy-based methods, statistical methods, transform domain
methods, and deep learning-based methods. The peak signal to noise ratio
(PSNR) and structural similarity index (SSIM) is the most commonly used
image quality assessment matrix. In recent years, CNN based models have
shown improvement in PSNR values for some benchmark datasets com-
pared to other best techniques today [25].
Image Registration in medical imaging is the process of combining data
captured from different modalities such as CT, MRI, SPECT, or PET to
obtain complete information for patient diagnosis. For example, in radio-
therapy treatment planning, a combination of CT and MRI is used as MRI
is used for the delineation of tumor tissues. In contrast, CT is used for accu-
rate radiation dose computation. Images from different modalities can be
taken at different times, from different viewpoints, and by different medi-
cal imaging equipment settings. The image registration method appropri-
ate for a given task depends on input data types, acquisition process, and
required accuracy and sensitivity to errors [27]. The images captured from
various imaging modalities are misaligned even if they represent the same
anatomical area, i.e., micro CT and IUVS represent the same coronary seg-
ment with different geometric coordinate scaling. The image registration
method’s necessary steps are feature detection, feature alignment from the
test image and the reference image, followed by parameter estimation by
mapping functions, image transformation, and image resampling [28].
Provided below Figure 11.2 is the simple design of a neural network. In
the next section, we will dwell more into deep learning and classify various
practical examples of it in two broad categories.

Hidden layer

Input layer
Output layer

Figure 11.2 Basic neural network design.

 Deep Learning in Medical Image Analysis 303

11.2 Deep Learning Models and its Classification

In this section, fundamental concepts from the literature of basic deep
learning models along with their architecture and broad categories are
discussed.

11.2.1 Supervised Learning

In Artificial intelligence, supervised learning is a branch of systems and
algorithms that provides a predictive model hinged on labeled data out-
comes. Models are appropriately trained by a different learning algorithm
and are thus optimized. Some types of supervised learning networks are
discussed below in subsections.

11.2.1.1 RNN (Recurrent Neural Network)

The Recurrent Neural Network (RNN) is a model of neural sequence that
achieves excellent performance on tasks like speech recognition [30], lan-
guage modeling [29], and machine translation [31]. In 1982 a special kind
of RNN called John Hopfield discovered Hopfield Network. Then in 1986,
David Rumelhart developed the RNN we use today. This is also used for
image classification. RNN is a kind of neural network where node connec-
tivity is of the directed graph along with the time-related sequence giving
it a time-related dynamic structure. Derived from feed-forward neural net-
works, but unlike it, RNN can process variable-length sequences of input
through its internal state. The main feature of RNN is hidden layers. Here
we will discuss the architecture of the simplest version of recurrent neural
networks, i.e., Elman Network [32]. It is simple to implement and train.
The network, in the time frame, has an input layer x(t), hidden layer s(t),
output layer y(t). Input vector x(t) is formed by concatenating vector w
representing a current word, and output from neurons in context layers at
time t − 1.
An input layer, hidden layer, output layer are calculated as below:
x(t) = w(t) + s(t – 1)

s j (t ) = f (∑ xi (t )u ji )
y k (t ) = g (∑ si (t )v kj )
304 Biomedical Data Mining for Information Retrieval

Where f(z) is sigmoid activation function and g(z) is softmax function respectively,

1
f (z ) =
1 − e−z

e zm
g (z ) =
∑e k
zk

For initialization, s(0) vectors can be given any small value because
while processing enormous data, initialization value does not matter. The
size of the vector x depends on the size of input data and the size of the hid-
den layer s(t) is usually around 50–500. This hidden layer s(t) depends on
data size. The networks are trained in epochs where data are represented
from training corpus sequentially.
Weights are initialized to random Gaussian noise with small mean and
variance, and learning rate α is also initialized. The training is carried out
using a standardized backpropagation algorithm along with stochastic
gradient descent is used for training. If the log-likelihood of the data is
increased, training is carried out in a new epoch else learning rate is halved.
Within 10–20 epochs, the network is trained. The softmax equation makes
sure that the distribution is valid. With each training, step weight is calcu-
lated, and weights are updated using a backpropagation algorithm. In the
end, the optimization algorithm is applied depending upon the data to get
the desired output.

11.2.2 Unsupervised Learning

Unsupervised machine learning algorithms dig out patterns from a data-
set without reference to known or labeled outcomes. Unlike supervised
learning, the learning method cannot be directly applied. They discover
the unknown pattern in the data. Some of the unsupervised learning net-
works are discussed below.

11.2.2.1 Stacked Auto Encoder (SAE)

SAE is a kind of generative model of autoencoders (AE). It has the property
of dimension reduction and is thus widely used to resolve applied prob-
lems. Autoencoder is a sort of unsupervised learning structure consisting
of three layers: an input layer, a hidden layer, and an output layer. The auto-
encoder is made up of two bits: encoder and decoder. Both of these parts
 Deep Learning in Medical Image Analysis 305

need to be trained. An encoder is handed down for conversion of the input

data into a hidden model, and the decoder is referred for rebuilding input
data from the hidden model. Given the unlabeled input dataset, {( xn )nN=1 }
where xn ∈ Rm1 and hn denote the hidden encoder layer obtained from xn
hereas yn is the decoder vector reconstructed from the hidden layer [33].
The function used in the encoding process is given below:

hn = f (w1xn + b1)
Where,
w1: Weight matrix used for encoder
f: Encoder function
b1: Biased vector for encoder

Similarly, below is the function used in the encoding process:

yn = g (w2hn + b2)
Where,
w2: Weight matrix used for decoder
g: Decoder function
b2: Biased vector for decoder

To optimize the network and minimize the reconstruction error param-

eters are set. Error is calculated by the formula below with L representing
the loss function:

1
∑
N
Φ (θ) = θ, θ′arg min L( x i , y i )
n n =1

The structure of SAE can be described as stacking of all autoencoders

into N hidden layers using an unsupervised algorithm learning features
layerwise and then using a supervised method to fine-tune it. The complete
training process of SAE can be divided into three steps:

(1) Training of the first autoencoder by feeding input data, then

obtaining the learned feature vector.
(2) The feature vector of the current layer is fed as an input for the
next layer, and this process repeats over the entire training.
(3) With the completion of training of all the hidden layers, the
cost function is minimized using the backpropagation algo-
rithm, and weights are updated to reach fine-tuning.
306 Biomedical Data Mining for Information Retrieval

Dropout and ReLU are used to overcome the overfitting problem, which
is present when the data fed is small. This way, accuracy is maintained, and
errors are minimized.

11.2.2.2 Deep Belief Network (DBN)

Deep Belief Network (DBN) is a generative graphical model. Alternatively,
it can be described as a class of deep neural networks, made up of mul-
tiple layers of hidden units, with connections between the layers but not
between units within each layer [34]. DBN can be visualized as a compo-
sition of a simple, unsupervised network like Variational Autoencoder or
Deep Boltzmann Network. The hidden subnetworks layer acts as a visible
layer for the upcoming layers. The training method for RBMs proposed by
Geoffrey Hinton for use with training “Product of Expert” models is called
contrastive divergence (CD).
A restricted Boltzmann machine (RBM) is a single-layer undirected
graphical model with a visible layer and a hidden layer. It assumes symmetric
connectivities between visible and hidden layers, but no connections among
units within the same layer, thus, it acts like an autoencoder. Because of the
symmetry of the connectivities, it can generate input observations from hid-
den representations. Training of parameters is generally carried out by a CD
algorithm to maximize the log-likelihood of observation and data. DBN is
a stacked RBM hierarchically. The first two form the undirected generative
model, and the last two layers form the directed generative model.
A DBN has visible layer v and several hidden layers h(l) where l represents
the hidden layers [1,......, L]. Therefore the joint probability distribution of
observed visible unit s and L hidden layers given by the following equation.
L −1
 
l L
P(s , h ,............., h ) = 

∏
l =0
P(hl | hl +1 ) P(h L −1 , h L )


Where,
P (hl |hl+1): Conditional probability distribution for the units of
layer l given the layer l + 1,
P (hL−1, hL): Joint probability distribution of the units in layers
L − 1 and L.

Greedy layer-wise pretraining schemes can be applied to learn and train

parameters in the following steps [35]:

(1) Put s = h(0) and train the first layer as RBM.

 Deep Learning in Medical Image Analysis 307

(2) Input for the next hidden layer is obtained from the previous
layer with the mean activation of P(h(1) = 1|h(0)) or samples
drawn from P(h(1)|h(0)) which is used as observation for
the second hidden layer.
(3) The second layer is also trained as an RBM; taking mean
activation or sample from the trained example.
(4) Repeat steps 2 and 3 until all the layers are trained until the
required data is met.

Log-likelihood of observed data is maximized using the Wake sleep

algorithm. After that the whole DBN is correctly trained, no further train-
ing is required.

11.2.2.3 Deep Boltzmann Machine (DBM)

Boltzmann Machine (BM) is a sort of stochastic recurrent neural net-
work model. It is named after Boltzmann statistics from statistical phys-
ics used for sampling functions. Because of the same reason, it is called
the energy-based model. It was invented by Geoffrey Hilton and Terry
Sejnowski in 1985 [36]. The Deep Boltzmann Machine is multiple BM
stacked in one. It works as a single system, capturing complicated fea-
tures among each layer in finding higher-order correlations in the layer
below. Like DBN, DBM also learns internal representation with increas-
ing complexity, which helps solve cognitive problems. Secondly, high-
level representations can be constructed from an ample of unlabeled
sensory inputs, and limited labeled data can then be used to fine-tune
the model for different tasks. Finally, unlike DBNs, the approximate
inference procedure and an initial bottom-up pass, assimilate top-down
feedback. This allows deep Boltzmann machines to better disseminate
uncertainty about, and hence helps the network to deal robustly with
ambivalent data.
Consider a two-layer Boltzmann machine with no within-layer connec-
tions. The energy of the state {s, h1, h2} is described as:

E(s, h1, h2;θ) = −sTW1h1 – h1TW2h2

Where,
θ = {W1, W2} are the model parameters representing visible to visible or
hidden to hidden interference.
The probability that the model is allocated to a visible vector s is:
308 Biomedical Data Mining for Information Retrieval

P(s ;θ ) =
1
Z (θ ) ∑exp(E(s, h , h ;θ ))
h1 ,h2
1 2

Logistic functions give the conditional probability distributions over the

visible and the two sets of hidden units:

P ( h1j = 1| s , h 2 ) = σ 
 ∑W s + ∑W
i
1
ij i
m
h 
2 2
jm j


P ( hm2 = 1| h1 ) = σ 
 ∑W h 
i
2 1
im i

P ( si = 1| h1 ) = σ 
 ∑W h 
i
1
ij j

The learning procedure for the general Boltzmann machine is applied

to maximize log-likelihood, but it is slow, mainly when there are several
hidden layers with complexity. Sometimes the greedy approach is used to
maximize the likelihood and to reduce the error [34]. After that, the net-
work is trained to solve cognitive problems with maximum accuracy.

11.2.2.4 Generative Adversarial Network (GAN)

In 2014, Ian Goodfellow and his colleagues invented the Generative
Adversarial Network (GAN). GAN consists of two neural networks, a gen-
erator, and a discriminator, respectively. As the name suggests, generative
networks generate data while discriminative networks evaluate them [37].
The external generator network is to increase the discriminative network’s
error rate. This means fooling the discriminator from discriminating
against the new data generated from the existing data. An existing known
dataset serves as the initial training data for the discriminator. Its training
involves feeding samples from the training dataset until it attains accept-
able accuracy. The generator gets trained until it fools the discriminator.
Typically the generator is fed with randomized input that is sampled from
a predefined multivariable normal distribution. After that, data synthe-
sized by the generator is evaluated by the discriminator. Backpropagation
is applied in both networks so that the generator produces better images,
while the discriminator becomes more skilled at identifying synthetic
images. The generator is usually a Deconvolutional neural network, and
the discriminator is a Convolution neural network (CNN).
 Deep Learning in Medical Image Analysis 309

input output output input layer

layer input hidden hidden and
layer hidden layers layer layer layer layer output layer
(a) (b) (c)

Generated Fake
Data
Noise
Real Real
Data
input layer
hidden layers and discriminator
output layer generator

(d) (e)

Figure 11.3 Various neural network architectures. (a) Recurrent Neural network, (b)
Autoencoder, (c) Restricted Boltzmann Machine, (d) Deep Belief Network, (e) Generative
Adversarial Network. x, y, h represent input, output and hidden layers (Inspired by: Ker
et al. [40]).

GAN is widely used in the fashion industry to produce new images of

models. They are used to improve astronomical images and simulate grav-
itational lensing for dark matter research [38]. GANs can reconstruct 3D
models of objects from images. In medical, it also has several emerging
uses, and one such is low CT image denoising [39]. Provided in Figure 11.3
is the architecture of various neural networks.

11.3 Convolutional Neural Networks (CNN)—

A Popular Supervised Deep Model
Convolutional Neural Networks are trained on large medical image data-
bases using supervised learning. It is used extensively for medical image
denoising, segmentation, and classification so that accurate image analysis
can be done. It is difficult to find accurate information for the diagnosis of
disease from medical image modalities like MRI, PET, Ultrasound, X-ray,
which are corrupted by noise.
Hubel and Weisel introduced the concept of CNN in 1962; when the
research was on monkeys and cats’ visual cortexes [41]. They observed
that visual cortexes of monkeys and cats contain neurons that individu-
ally respond to the visual field. They discovered that the cat’s visual cor-
tex consisted of simple cells and complex cells, which were invariant when
distorted inputs were arranged hierarchically. This research gave the early
inspiration to create a model of automated vision systems based on central
310 Biomedical Data Mining for Information Retrieval

nervous system attributes. The idea of the CNN receptive field is taken from
this. The ‘Neocognitron model’ was designed by Kunihiko Fukushima in
1980, and was motivated by Hubel and Weisel’s research. Pattern recogni-
tion was done in the neocognitron model with the help of convolution and
down-sampling layers [42]. The successful implementation of LeNet-5 by
LeCuN, which was a handwritten digit classification network, laid the foun-
dation of the basic CNN model, which is being modified and used for med-
ical image classification [14]. After that, many pioneer CNN classification
networks were designed in the ImageNet Large Scale Visual Recognition
Challenge (ILSVRC), a software design competition for an object detection
run by ImageNet annually. In 2012, the AlexNet network outperformed all
the other image classification methods and garnered the research commu-
nity’s considerable attention in the design of deep CNN’s. The other famous
image classification CNN’s are VGGNet, GoogleNet, ResNet, ZFNet [43].
Compared to other deep learning neural networks, there are several
advantages of CNN, specifically when input is image data. The hypothesis of
weight sharing between neurons in CNN leads to a reduction of the network
complexity and improves generalization as compared to the artificial neural
network. CNN’s can be trained easily by backpropagation as sparse connec-
tions are present in each convolution layer. The application-­oriented size-
able medical image databases are available, which can be used for training in
graphics processing units (GPUs) for efficient and faster computation. CNN
possesses higher representation capability, which provides better image reso-
lution performance compared to traditional sparse representation. In sparse
representation, 2D structural information, i.e., information pertaining to
the pixel dependency in the local adjoining neighborhood region, is lost.
The sparse dictionaries are formed with vectorization of the image matrices
leading to two-dimensional structural information loss. On the other hand,
CNN is efficient in retaining two-­dimensional structural information during
the training and testing process. The convolution operator with 2D masks
inherently includes the local neighboring image pixels. Transfer learning is
applied to CNN’s in which pre-trained networks are modified by changing
input dataset, no. of layers, modification in the loss function to design a new
network which inherits many features of the pre-trained network.

11.3.1 Architecture of CNN

CNN comprises an input layer, output layer, and several intermediate
hidden layers that transform input data through a series of various linear
and non-linear transforms done by appropriate mathematical functions.
The feature maps are produced as an output of the input layer and each
 Deep Learning in Medical Image Analysis 311

intermediate hidden layer. The previous layer’s input data is multiplied

by a convolutional filter matrix of a specified size and weight to extract
a particular feature of the image. So, the output of hidden units is certain
functional processing of input data processed by convolutional filters. The
linear convolutional operator output is processed by non-linear activation
functions such as sigmoid, rectified linear unit, tanh.
The general mathematical equation depicting CNN operation is:

X ji = f 
 ∑ iMj
xil −1 ∗ w lji + blj 

(11.1)

Where Mj represents the selection of the input feature map, xil −1 is the output of
the previous layer i.e. feature map, w ijl the weight of the convolution kernel of the
lth layer, f is the activation function and, blj is the bias of the lth layer.
The typical CNN architecture for training a 256 × 256 input image
with a supervised learning objective function approach is given in Figure
11.4. The objective function or loss function is used for optimization for
training CNN for a particular image processing application. The different
operations taking place in CNN architecture are convolution, pooling, and
non-linear transformation using the activation function.

Convolution:
The convolution filter consists of weight kernels that detect a particular fea-
ture of the image. The convolution filter of fixed size is translated all over
the image, and output is matrix multiplication of convolution filter weights

Input Image 64 Feature Maps 64 Feature Maps O/p Image

256×256 256×256 256×256 256×256
(RGB) (RGB)

64 Conv filters 64 Conv filters 3 Conv filters

3×3×3 Bias 3×3×64 Bias 3×3×64 Bias

Desired Output
Image
256×256 Obj Func. c(.)=
(RGB)

Figure 11.4 An illustration of a typical CNN architecture for pixel RGB images (includes
the training objective function).
312 Biomedical Data Mining for Information Retrieval

3 0 1 5 0 3 0 3

2 6 2 4 3 0 3 0 -3
–1 0 1
2 4 1 0 6 1 4 1
–2 0 2
3 0 1 5 0 3 0 2

2 6 2 4 3 2 3 0 –1 0 1
2 4 1 0 6 2 1 1
(b)
2 6 2 4 4 0 3 1

2 4 1 0 6 1 6 6

(a) (c)

Figure 11.5 (a) Input Image, (b) Convolutional filter and (c) Convolved output value for
the highlighted pixel.

with the corresponding part of the input image matrix. The convolution
filter slides over the entire input image as shown in Figure 11.5. The num-
ber of pixels in which convolution filter slides horizontally or vertically to
perform convolution over the whole image is called a stride. The convolu-
tion filter’s output is termed as a feature map because it extracts particular
image features like edges and artifacts. Sometimes, zero paddings are also
performed, i.e., image size increases by padding zeros around the image to
get the desired size feature map.

Pooling:
The pooling operator is applied to a feature map for spatial resolution reduc-
tion. The feature map becomes spatial invariant to input data distortion and
translation when its spatial resolution is reduced. Initially, average pooling
was used in CNN models, i.e., the average of all input pixel values of the local
neighborhood were propagated to the subsequent layer. In the recent mod-
els, maximum pooling operation is done, which propagates maximum value
within an image neighborhood to the subsequent layer. The max-­pooling
operator selects the most considerable pixel value in the given receptive
field. Figure 11.6 depicts max-pooling and average pooling process on the
input image of size 4 × 4, convolution filter of size 2 × 2 with a stride of two.
The max-pooling selects the maximum pixel value in 2 × 2 sub-window,
whereas average pooling calculates each sub-sampled 2 × 2 window.

Activation Function:
The activation function transforms convoluted output by performing a
non-linear transformation. The predominantly used activation functions are
rectified linear units, tan hyperbolic function, softmax function, sigmoid.
 Deep Learning in Medical Image Analysis 313

Average
15 9 11 5 10 5

11 5 8 6 5 8

8 7 3 17

2 3 7 3 Max 15 8

8 17

Figure 11.6 Average versus max pooling with a stride of 2.

11.3.2 Learning of CNNs

The different learning algorithms used for training the CNN are conjugate
gradient methods, Quasi-Newton Methods, Levenberg–Marquardt Method,
first-order hybrid training methods, Least Squares Method, and Quasi-
Newton Methods [44]. The training methods are categorized into objective,
and learning function approaches. The minimization of the loss function or
the objective function is the solution of the machine-learning problem in
the objective function approach. On the contrary, in the learning function
approach, a parametric function is used to find a solution to the inverse prob-
lems in the medical imaging field. A parametric function is the solution of the
regularized minimization problem. Loss function (Cost Function) comprises
parameters of the neural network, input, and desired output. It is formulated
in such a manner that its minimization leads to the desired output. The most
commonly used loss function in supervised learning is the mean square error.

1
∑ ∑
P M
E(k ) = ( zij − tij ) k = 1 to Sk .
2
(11.2)
P×M j =1 i =1

Where M and P are the number of rows and columns in image, Sk is the
number of training iterations, zpm and tpm represents an actual and
desired image.
The stochastic gradient descent method is utilized in optimization, i.e.
minimization of this mean square error. The gradient descent method iter-
atively modifies the weight (w) from w(t) to w(t + 1) with the weight upda-
tion equation given as follows:

∂ E( k )
w(t + 1) = w(t ) − η (11.3)
∂ w(t )
314 Biomedical Data Mining for Information Retrieval

where ƞ is the learning parameter. The general learning equation of CNN

to solve the inverse problem in imaging given in Ref. [45] as:

∑
P
Rlearn = Rθ arg arg min C ( x j , Rθ ( y j )) + g (θ) (11.4)
j =1

The ground-truth image pairs (xj,yj) are available in training datasets,

where xj is a clean image, and yj is its noisy version. C is cost function
or loss function, R depicts mathematical function encrypted in CNN to
extract denoised image xj, g is regularization operator which assists in pro-
ducing solution which complements the input image xj prior knowledge,
denotes CNN parameters.

11.3.3 Medical Image Denoising using CNNs

The medical images such as MRI scan, CT, PET, mammograms, X-ray, and
ultrasound are often corrupted. This includes different levels of Gaussian
noise, Rician noise, salt and pepper noise, mixed noise during transmis-
sion and acquisition. The corruption of images with noise complicates the
diagnosis, analysis, and treatment process. The objective of medical image
denoising is denoised image extraction from different medical image
modalities for proper diagnosis.
In the recent era, CNN has shown significant improvement in peak sig-
nal to noise ratio (PSNR) compared to other benchmark image denoising
techniques [25]. The PSNR is used as an image quality assessment metric
in most of the image denoising research works. The feed-forward denois-
ing CNN (DnCNN) is based on residual learning and is a widely popular
network for image denoising [26]. Although it was initially trained using
Berkeley Segmentation Dataset (BSD-500), the same architecture was
used to denoise medical images by changing the training and testing data-
set [46]. DnCNN can enhance the image quality of three general inverse
imaging problems, i.e., Gaussian denoising for both known and unknown
input standard deviation, image deblocking (JPEG) with the quality fac-
tors of different values, and image super-resolution with various upscaling
parameters.
In DnCNN, residual learning in coordination with batch normaliza-
tion fasten up the network training procedure and reduces the standard
deviation level, i.e., Gaussian noise level. Residual learning solves the per-
formance degradation condition to a certain extent, i.e., training accuracy
deteriorates as the network length increases [47]. The output is residual
 Deep Learning in Medical Image Analysis 315

noise rather than a denoised image in case of a residual learning approach.

Batch Normalization is used to improve the training efficiency of stochastic
gradient descent method; it attenuates the internal covariate shift by intro-
ducing a normalization step and a scale and shift step before the activation
function intermediate layers of CNN [48]. Blind Gaussian denoising, i.e.,
removal of Gaussian noise of undefined standard deviation level, is also
done by the DnCNN network. Figure 11.4 illustrates the block diagram of
DnCNN architecture.
There are three kinds of layers in DnCNN of depth D:
(i) Conv + ReLU (1st layer): Number of filters = 64; Filter size
= 3 × 3 × c (gray scale c = 1 and RGB image c = 3); Number
of feature maps = 64; Activation Function = Rectified lin-
ear units (ReLU, max(0)).
(ii) Conv + BN + ReLU (2 (D − 1) layer): Number of filters =
64; Filter size = 3 × 3 × 64; Number of feature maps =
64; Inclusion of Batch Normalization before activa-
tion; Activation Function = Rectified linear units (ReLU,
max(0)).
(iii) Conv (last layer): Number of filters = c; Filter size = 3 ×
3 × 64.

Stochastic Gradient Descent does the training of the benchmark

DnCNN image denoising network. The average means a square error of
the desired residual images and predicted CNN image output is procured
as the objective function (loss function) for learning the network parame-
ters denoted by .
The CNN variants to denoise the medical images are derived from the
DnCNN model. They change training and testing datasets, no. of layers in
the architecture, loss function, and applying a transfer learning concept
to the existing network. Multi-channel DnCNN uses residual learning for
Rician noise reduction from MR brain images of the IXI and BrainWeb
dataset [49]. CNN-DMRI network uses encoder-decoder architecture
with local and global residual learning for MRI denoising [50]. PET image
denoising of brain and lung PET scans is done using residual learning-­
based pre-trained CNN with the perceptual loss [51]. A deep CNN archi-
tecture inspired by the DnCNN model is designed for denoising low-dose
computed tomography perfusion data. The training input and output are
low-dose cerebral blood flow (CBF map) and the high-dose CBF map,
respectively [52]. Vessel-edge-preserving convolutional neural networks
(CNN) denoise simulated low x-ray dose percutaneous coronary interven-
tion (PCI) images formed by the addition of artificial noise to high-dose
316 Biomedical Data Mining for Information Retrieval

images [53]. Its objective function is optimized to achieve the edge-­

preserving effect of vessel walls.

11.3.4 Medical Image Classification Using CNN

In medical image classification, target lesions are categorized or classified
into a different number of classes. CNN is a deep learning network that
extricates the features of an image and utilizes them for image classifica-
tion. E.g., CNNs are frequently used for classifying lung nodules on CT
images, which are divided into benign and malignant classes. Similarly, a
tumor on breast cancer histopathology images is also classified as benign
or malignant by appropriate training of CNN [54]. CNN can be used for
automatic glaucoma assessment using a fundus image as shown in Figure
11.7 [55].
The convolutional filters denoted by (Blue Squares) are applied to a
small receptive field of the input image but detecting at each pixel posi-
tion of the image and always covering through the whole image depth. The
resulting convolved output of the input image is known as feature maps,
further analyzed by convolution or pooling layers at each of their surface
positions. The number of feature maps generated equals the number of
convolution filters of CNN used for a particular input image. Each fea-
ture map corresponds to the output for feature extraction—the activation
function changes the output into probabilities of input medical image
belonging to different classes. The correct label joins the training dataset’s
predicted classes to calculate loss using object function like cross-entropy.

Deep Convolutional Neural Network

Softmax True
Function Label
Prediction

Input Image Predicted

Classes
Feature extraction through Global Pooling,
convolutional layers and pooling Flattening, and
Fully connected
filters cascade layer
Loss

Back Propagation: Weight Update for

Convolutional layers and dense layers

Figure 11.7 Basic CNN architecture for image classification [55].

 Deep Learning in Medical Image Analysis 317

The learning methods (ex. Stochastic gradient descent) are utilized to

update the weights of the convolutional kernel and fully connected layer
based on losses propagated through the network.

11.4 Deep Learning Advancements—A Biological

Overview
Applications of deep learning, especially CNN, have seen a great explo-
sion since the last two decades in the field of medical image analysis.
Feature extraction, segmentation, registration, classification, indexing,
retrieval, and visualization (or detection) are all parts of the automated
medical image analysis. Here we will try to classify significant advance-
ments biologically made in this field. Biologists tend to visualize bigger
and complex organisms like humans as a collection of small anatomical
structures at every stratum. Usually, the starting point in such studies is a
single cell, the basic structure of life, capable of independent existence. A
group of tightly packed, coordinating cells forms tissue, then some tissue
performing the same function and sharing a common origin, forms an
organ, and so on.
The first significant breakout in the field was reported by Carpenter
et al. [56] in 2006, named ‘CellProfiler,’ which aimed towards the automated
measurement of various subcellular and cellular features. We will breeze
through some of the applications that this work suggested in the next two
sections and see how it has evolved since then. Each section would provide
an extensive review of the current advancements at that level and end with
a short discussion for future directions. The last two, tissue and organ sec-
tions are also supplemented with the detection and diagnosis of various
diseases. This will help readers to identify real-life medical applications of
deep learning algorithms.

11.4.1 Sub-Cellular Level

Subcellular space is everything enclosed within the boundary of a cell.
This space includes a fluidic protoplasm in which all the cell organelles
exist, working in harmony with each other. The significant deep learning
advancements for medical image analysis at the subcellular level are mostly
segmentation or classification based.
Neural networking based deep learning algorithms traces subcellular
delocalization through images captured by high-throughput fluorescence
screening (HTS) devices. Subcellular delocalization is a measure of relative
318 Biomedical Data Mining for Information Retrieval

spatial variability, as the function of intensity and time for a given probe.
The interconnected layers of algorithm scan for logical patterns invariance
and compute the subcellular spread [57]. These scans locate any mislocal-
ized protein among cellular compartments, which can trigger the onset
of diseases such as Alzheimer’s or cancer or some cardiovascular disor-
ders [58]. One of the earliest studies by Boland and Murphy [59] used a
back-propagation neural network (BPNN) on the HeLa cell lines for the
quantitative estimation of protein localization patterns. In 2008, Li et al.
[60] adopted the first automated approach for subcellular delocalization
of proteins using Human Protein Atlas (HPAs) reference, which analyzed
over 3,000 images with 87.5% accuracy. Tahir et al. [61], used a hybrid
SVM-SubLoc method to achieve more than 99% correctness over the data-
set. Like the earlier one, numerous tools and methods have been reported
in the last decade, with ever-improving accuracy and applicability. One
recent work (Yang et al. [62]) uses the MIC_Locator model with the lat-
est, multi-labeled dataset, and frequency features to outperform others
in the field. Many such works have also used yeast as a model organism
(Pärnamaa and Parts [63]; Kraus et al. [64]).
Another application of deep learning that uses HTS devices is drug dis-
covery. Mapping and assessing interactions between drug molecules and
their target sites form an essential component in the drug discovery pro-
cess, called ligand–protein interactions. Many studies have examined sub-
cellular perturbations in response to a specific drug and compared diverse
drug varieties for optimum response [65, 66]. An interesting approach is
to use unsupervised machine learning or supervised (ML) to screen for
phenotypic changes at cellular/subcellular level in response to a particular
stimulus like a drug molecule, thus helping in extensive testing before clin-
ical use [67]. A 2018 study used pre-trained deep CNN networks to map
cellular action mechanisms in response to a drug molecule with over 95%
accuracy [68].
Karyotyping also effectively uses neural networks - A process of describ-
ing shape and number of chromosomes in a cell. Chromosomal abnormal-
ities arise either due to altered numbers or/and structure of chromosomes
and can develop in fatal genetic diseases like leukemia or even cancer. A
very recent study proposes the use of a two-stage chromosomal classifica-
tion method in which a Q-banded dataset with 119 chromosomal images
and ANN-MLP classifier is used to obtain 88% classification accuracy [69].
Before this, many such efforts in the field involving segmentation, feature
extraction, and classification of strained chromosomal images were made.
These automated techniques can aid cytogeneticists with high accuracy
and less effort.
 Deep Learning in Medical Image Analysis 319

Chromosomal architecture is generally localized within the nucleus

unless the cell is dividing. Deep learning is immensely used in nuclei
detection in various histopathology images. Nuclei count and morphology
detection, along with segmentation and classification, are critical for eval-
uating a disease’s existence and severity. A study reported that intrusion of
lymphocyte nuclei is often correlated with patient survival and recovery
after breast cancer [70]. The CoNNACaeF model, which uses breast cancer
nuclei segmentation and detection network, produced over 80% accuracy
in training cancer detection dataset [71]. Nuclei pleomorphism also helps
in other types of cancer sorting and diagnosis. In 2016 an automated tool
that uses spatially constrained convolutional neural networks (SC-CNN)
excluding the segmentation process was formulated for quantitative analy-
sis of whole slide images (WSIs), leading to a better understanding of colon
cancer [72].
There are many other advancements in biomedical research. They may
not have immediate applications currently but can evolve in a great poten-
tial future for the medical field. One of them is ‘In silico Labeling’ (ISL).
The approach here is to eliminate the need for labeling a moiety to see it.
It uses deep learning algorithms to predict nuclei labels, other than cell
type and cell states detection. Similar work was done using CNN based
U-Net architecture to predict label-free 3D images of even subcellular
structures with high precision [73]. These techniques can be the future of
low cost, fast and accurate biomedical imaging that can quickly be done
on even living cells. Recently in 2018, an attempt was made to classify 13
major organelles inside a cell using protein localization imaging data, with
a fully convolutional neural network (FCN) [74]. The overall accuracy of
the model was reported low, although being current art-of-the-state. The
dyes were used to image major cell organelles, followed by the applica-
tion of automated computer-based profiling tool ‘CellProfiler’ to contrast
cellular/­subcellular morphologies based on given stimuli. This approach
can be effectively used for new drug discovery in the future.

11.4.2 Cellular Level

There have been tons of advancements in terms of applications for deep
learning in medical imaging at the cellular level. Pre-processing, segmen-
tation, detection are some common steps usually done here, followed by an
analysis of cellular morphologies. However, even before, an image dataset
needs to have quality control. Let us see how this happens.
Cellular imaging is an essential diagnostic tool in modern medicine.
Furthermore, microscopy is the heart of cellular imaging, but low quality
320 Biomedical Data Mining for Information Retrieval

and out of focus images are headaches for medical examiners, especially
when dealing with incredible large datasets. To fix this quality control or
technically, ‘Pre-preprocessing’ is done, which is imperative to compensate
for the unevenness of data and aid in further analysis. Autoencoders are
one great example of this, which are extensively used for differentiating
noise from accurate data. One of the pioneering work was done by Su et al.
[75] using stacked denoising autoencoders (sDAE) to detect shape vari-
ations and overlapping images of cell fragments and also perform effec-
tive cellular segmentation. Following this [76], a U-Net architecture was
presented to improve the high-quality fixation of 3D isotropic resolution
in fluorescence microscopy. Wang et al. [137] went a step ahead to scale
up the resolution of full-field fluorescence 10×/0.4NA microscopic images
to 20×/0.75NA objective. This employed GANs (Generative Adversarial
Networks) for the purpose. Autoencoders were used by Janowczyk et al.
[77] for strain normalization to standardize the color distribution of the
test images that were to be processed by computer-aided diagnostic (CAD)
algorithms. CNN also acts as a savior in quality control. Yang et al. [78]
used deep neural networks for classifying blurred microscopic images with
high accuracy and precision. They knowingly trained their model on 384
defocused stained nuclei images of U2OS cells to make the system more
robust, achieving over 95% accuracy.
Classification frameworks have been effectively used in the diagnosis
of various autoimmune diseases. Gao et al. [79] used an automated CNN
based approach for classifying HEp-2 (Human Epithelial-2) cells. HEp-2
cells act as a substrate for antinuclear antibody (ANA) screening. It is a
good indication of developing autoimmune diseases like scleroderma,
autoimmune hepatitis. Indirect immunofluorescence (IIF) is the diagnos-
tic tool used to classify cells based on intensity and type of fluorescence in
the staining pattern. Ref. [80] is the state-of-the-art technology that uses
ten optimum layered ANN without data augmentation for over 96% accu-
racy. Sickle cell anemia also uses classification efficiently for its diagno-
sis. Ref. [81] is the current best model that classifies erythrocytes, among
other blood cell types, into a circular (healthy) and elongated (sickle cells).
This model achieves over 99% accuracy in the classification process using
DL algorithms. Many other erythrocyte classifications are often done to
study the morphology profile of red blood cells (RBCs). Models generally
use deep CNN to classify RBCs into different morphological classes. Since
RBCs carry oxygen throughout the body, the study of their morphology
serves as an indication of numerous potential diseases. CNN is used to
detect and classify inflammatory cells such as neutrophils and lymphocytes
with Hematoxylin and Eosin (H&E). High numbers of inflammatory cells
 Deep Learning in Medical Image Analysis 321

are an indication of a probable infection in the body. Another study [82]

used a CNN based approach to classify infected and healthy macrophages
in the human body. Model also classify the strain of the infecting patho-
gen (Mycobacterium tuberculosis) as mCherry-expressing Mm wild type
or mutant strain with deletion of the esxA-esxB operon (Mm(ΔEsxA:B)),
with about 90% accuracy. Again the study was based on the morphology
classification of considered macrophages. Wang et al. [83] utilized clas-
sification tools for creating cellular spatial maps to facilitate modeling of
growth patterns and cell interactions with the tumor microenvironment
or immune system, thus better-predicting diseases like cancer in very early
stages.
There are numerous applications of detection which are often cou-
pled with classification and segmentation. Like, for example, cell count-
ing. Counting is a very commonly used tool for disease diagnosis or to
observe the after-effects of any treatment. To eliminate the cumbersome
way of counting manually reported techniques [84] were used. It is a Fully
Convolutional Regression Network (FCRN) based tool that utilizes a spa-
tial cellular density map that even operates on arbitrary sized input data.
To show that these tools can be fast and accurate Ref. [85] accelerated the
process by 10,000 times, it claims to detect 108-pixel cell images in just 20 s.
This was achieved using CNN and asynchronous prefetching techniques to
reduce disk I/O time significantly. Subtype cell detection is also vital in
histopathological images. Wang et al. [83] aimed their work at accelerat-
ing cell detection and providing cellular subtype information of detected
cells using the DCNN framework. They utilized d-regularly sparse ker-
nels to surpass and ease redundant computational processes. Another such
work in 2014, Chen and Chefd’hotel [86] used an automatic immune cell
detection system. They first used algorithms to differentiate RGB image
patterns into meaningful biological channels, followed by the detection of
cells using CNN. Not only in humans, Dong et al. [87] adopted super-
vised learning-based CNN in detecting wide-field microscopic z-stacked,
zebrafish images. Zebrafish serves as a competent and accessible biological
model organism and is generally used for tyrosine hydroxylase-containing
(TH-labeled) cell detection.
Detection frameworks are effectively used in disease diagnosis too. One
such disease is Malaria, where earlier manual detection was neither reliable
nor fast or cost-effective. A protozoan parasite that uses female anopheles
mosquito as its vector, is held responsible for Malaria. About 228 million
cases of this disease were globally reported in 2018 with 405 thousand mor-
talities, as reported by WHO (Organization and Others 2019). Recently
many new studies used pre-trained CNN based DL models to self-discover
322 Biomedical Data Mining for Information Retrieval

hierarchical feature representation in the images of blood smear. The pro-

cess classifies infected and uninfected RBC cells, thus detecting the sever-
ity of the disease. The process includes feature extraction, followed by
segmentation and detection. Another diagnosis strategy that uses a detec-
tion framework is tumor detection. Circulating tumor cells (CTCs) are
promising biomarkers for cancer medicine. Zeune et al. [88] proposed a
CNN autoencoder multiclass classification network along with fluorescent
visualization techniques to detect and count circulating cells based on their
morphology and origin with an accuracy, specificity, and sensitivity of over
96%. The results from the model show that it can accurately predict the
overall survival of cancer patients.
Deep neural networks (DNN) are also helpful in mitosis detection, which
is a challenging task for the human eye. An estimate of mitotic products is
an indication of potential developing cancer in histology slides. In 2012 a
DNN trained on numerous literature images was used to detect mitosis in
breast cancer cells [89]. It directly worked on RGB data obtained from a
specific spot of the source image and analyzed it based on learned visual
features from the mitotic training dataset. Today, R-CNN and deep CNN
are used to detect mitosis in cancer histopathology images, which are both
fast and accurate. A study recently used this along with state-of-the-art
post-processing false positive removal, to achieve nearly 87 and 84% preci-
sion over ICPR 2012 and ICPR 2014 dataset respectively [90]. Another work
utilized both live microscopic image processing and CNN to detect mitotic
cells from their counterparts [91]. Also, many time-lapse based imaging is
reported for mapping the complete cell cycle during mitosis. The computa-
tional framework ‘CellCognition’ used in this work involves machine learn-
ing algorithms along with Markov modeling to detect different biological
states of a cell. Eulenberg et al. [92] also trained CNN for classifying cells in
different stages during the cell cycle, followed by non-linear dimensionality
reduction to reconstruct temporal progression of this cycle.
There are many futuristic approaches like unsupervised learning that
have become popular in recent years. Covering one of the most significant
limitations of supervised learning, Zhong et al. [93] proposed a temporally
constrained combinatorial clustering model for accurately predicting cellu-
lar morphologies using time-resolved images. The most prominent edge of
the technique was its independence over user-annotated training datasets.
The reported work could easily classify human cellular phenotypes, pretty
accurately. Although the unsupervised approach is currently limited to a few
examples at the cellular level, it will expand in the future. Another futur-
istic application is the multidimensional modeling of a living cell. This is
not something new, rather an improvement over many earlier studies that
 Deep Learning in Medical Image Analysis 323

Table 11.2 Some supplementary advancements in the subcellular and cellular

sections.
Application Principle References
Secondary protein CNN classifier Li et al. [60]
structure
Drug discovery Deep CNN Pawlowski et al. [95]
Quality control/ Regenerative random Paul et al. [96]
Pre-processing forest tree classifier
CNN Mao and Yin [97]
CNN Balkenhol et al. [98]
CNN and GANs Ouyang et al. [99]
Concentric circles Li et al. [100]
Malaria Detection CNN Rajaraman et al. [101]
Segmentation Deep algorithms Shirazi et al. [102]
CNN Van Valen et al. [103]
CTCs detection Support Vector Mao et al. [104]
Machine and CNN

have already attempted to build 3D cell models. While Chen et al. [94] have
developed a 5-D (x, y, z, t, λ) cell imaging model on dendritic cells (DC)
and T-cells of evolving mouse thymic cortex and lymph nodes. Model used
a two-photon laser scanning microscopy (TPLSM) technique for this. The
future improvements can be to fabricate these models more robustly by
closely simulating effects of the environment such as drugs or pathogens as
an example, for better applicability in medical diagnosis. Table 11.2 describes
some more DL-based applications at the subcellular and cellular level.

11.4.3 Tissue Level

The majority of tissue-level imaging advancements concerning deep learn-
ing have been made either in tumor/cancer detection and classification
or on glands segmentation. First, we must be clear about cancer as one
of the deadliest humankind diseases and tumors as one of its after-effects
resulting from the uncontrolled growth of cells in solid tissues like mus-
cles, bones, and organs.
324 Biomedical Data Mining for Information Retrieval

However, pre-processing is also a thing here. Ciompi et al. [105] have

demonstrated the importance of stain normalization (a Pre-processing
step) in training automated models using a DL based convolutional net-
work. The aim was to classify colorectal cancer tissue in H&E stained histo-
pathology images. Rivenson et al. [106] successfully attempted to improve
the resolving power of bright field images from Masson’s trichrome kid-
ney tissue sections acquired by a lower 40×/0.95NA objective to a higher
100×/1.4NA objective. Models use deep CNN for this purpose. Nowadays,
to promote portable and cost-effective medical imaging, mobile-phone
microscopy is gaining popularity. Deep learning-based algorithms boost
the resolution of images acquired by small portable devices and help in
image denoising and color-correction, thus entirely transforming mobile
captured tissue section images for medical diagnosis [106].
Gland segmentation is necessary because, unlike individual cells, FCN
models are not well suited for systems where individual objects are needed
to be identified, as a tissue section. Deep learning has been immensely used
in context with different glands and tissues. Xu et al. [107] used it for seg-
menting individual glands (instances) in colon histology images, using side
supervision guided CNN. Current best techniques [108] can even operate
on images with higher dimensions like 1,792 × 1,792 pixels and effectively
grade them using CNN. It encodes local symbols of tissue section images
into high dimensional features and then bundles them all into final pre-
diction. Similarly, Wang et al. [109] used the segmentation approach in
prostate cancer research on MRI scans using deep CNN. Model even went
a step ahead to prove why the DL approach is better than the non-DL one,
showing a significant Area Under the Curve (AUCs) difference of 0.14
between the two compared techniques. Recently, Bulten et al. [110] devel-
oped a DL model that aims to segment the epithelial tissue in H&E-stained
prostatectomy WSI (Whole Slide Images). They first trained a U-Net archi-
tecture for epithelial structures segmentation in pre-processed IHC slides.
They then applied the model on others to create a standard for reference,
which was used to train another U-Net on H&E slides.
Segmentation is a technique that has been effectively used at tissue
level for cancer and tumor detection. In two different studies, Ehteshami
Bejnordi et al. [109] showed effective use of DL algorithms in H&E stained
lymph node tissue sections for detecting metastases in breast cancer cases.
Litjens et al. [111] showed valid identification of breast cancer metastasis
detection and prostate cancer from biopsy specimens from lymph nodes
using DL algorithms. The patch-based adopted technique for both the above
studies was to image the complete slides (called Whole Slide Imaging or
WSI) into segments and then treat individual segments as one training set.
 Deep Learning in Medical Image Analysis 325

Similar approaches are used by many to investigate breast tumor prolifera-

tion using WSIs. Further, Cruz-Roa et al. [112] implemented a new efficient
adaptive method by only sampling relevant segments as an improvement
to this. Model used a High-throughput Adaptive Sampling (HASI) model
that uses probability gradient and quasi-Monte Carlo sampling along with
CNN deep learning classifiers. This increases both the accuracy and time
efficiency of the system. Another example of segmentation is [113], where
the authors used glioblastomas MR (Magnetic resonance) images to map
brain tumor tissue sections. Model used DNN algorithms that simulta-
neously extract global and local contextual features and claimed to have
accelerated the process about 30 times faster than others. These conven-
tional approaches are used for the classification and detection of various
types of tumors across the body.
Classifying tumors into malignant and non-malignant/benign is the next
important step for early cancer diagnosis. One such effort using CNN algo-
rithms was made by Bayramoglu et al. [114] to classify benign/­malignant
properties of a breast cancer tissue section. Models carried out their evalu-
ations and comparisons on the BreaKHis dataset, containing breast cancer
histopathology images. Turkki et al. [115] adopted a CNN based proce-
dure to determine the fate of breast cancer patients by classifying them
into low and high-risk categories. Model did this via morphology-based
feature extraction from stained tumor tissue images by incorporating
direct outcome prediction using a tissue morphology approach. A similar
direct prediction method was utilized by Mobadersany et al. [116] for pre-
dicting the survival chances of patients diagnosed with glioma.
Model organisms are often used for pre-clinical trials of various drugs
and therapies to optimize them for human use. Häring et al. [117] uti-
lized cyclic consistent generative adversarial networks (GANs) for the
segmentation of epithelial tissue from Drosophila melanogaster embryos.
Drosophila serves as a popular model organism for many biological stud-
ies. Model also compared their approach with other latest techniques like
Tissue Analyzer and U-Net, thus claiming their model to be the current
state-of-the-art. Recently, Asay et al. [118] introduced the ‘Lesion Image
Recognition and Analysis’ (LIRA) tool that employs CNN to effectively
classify seven main pathology features, including three different lesion
types from pulmonary tissues from mouse tuberculosis infection models.
This could be a more significant aid for experimental pathologists.
Global challenges like Bandi et al. [119] provide an excellent platform
for future improvement. They recently organized ‘The CAMELYON17
Challenge’ in collaboration with IEEE international symposium. The
topic was ‘Biomedical imaging 2017 conference’, held at Melbourne. This
326 Biomedical Data Mining for Information Retrieval

challenge was to develop algorithm based detection of individual potential

metastatic tissues to classify personal lymph node status. Then all the par-
ticipant algorithms were evaluated based upon the large test set. Such com-
petitive challenges can lead to some revolutionary developments. Marker/
label-free tissue section imaging is one futuristic approach that is currently
being worked upon using GANs and other unsupervised ML techniques.
The future in the field is to develop better algorithms that are both accurate
and time-efficient, which can deal effectively with issues like overlapping
cell segmentation in the tissue section.

11.4.4 Organ Level

Organs are not just a structural and functional collection of tissues;
instead, they are part of a living body. Moreover, a living body is continu-
ously exposed to various traumas and diseases, be it as small as a wound or
incurable Alzheimer’s. However, deep learning can serve in diagnosis for
both and also for many other deformities like these. Here in this section,
we present a comprehensive organ wise in-depth learning advancement
overview for the human body. Although most of the techniques reported
here are supervised learning-based being more applicable and accessible
today, Raza and Singh [120] has provided an overview of some unsuper-
vised learning tools as a more futuristic automated approach. We would
also occasionally discuss some futuristic directions along the way.

11.4.4.1 The Brain and Neural System

Significant advancements can either be classified as a brain lesion, whole
tissue, and subcortical structure segmentation or for detecting diseases
like Alzheimer’s and Parkinson’s. Most of the brain segmentation tasks
broadly undergo preprocessing, classification, and postprocessing steps.
Preprocessing involves removing unwanted non-brain tissues of the head
acquired by MRI scans, also called ‘Skull stripping,’ [121] which are mostly
CNN based automated tools. Another preprocessing step involves intensity
correction (created by inhomogeneous magnetic fields and MRI coils). For
which bias-field correction [122] or nonparametric intensity normaliza-
tion [123] are deployed. Intensity normalization is also used in removing
intraclass variability. Image registration is generally done after the pre-
processing step, to convert different MRI modalities into common coor-
dinate space [124]. Data argumentation and patch extraction are the last
two steps before feeding data into a network. Classification often employs
two different strategies. One is a single path architecture [125]. It proposed
 Deep Learning in Medical Image Analysis 327

high-resolution 3D CNN for segmentation of 155 neuroanatomical

structures using relatively more straightforward, compact algorithms on
brain MR images. Another such work by Zhang et al. [126], was aimed
to segment the infant brain using 2D images from white matter (WH),
grey matter (GM) and cerebrospinal fluid (CSF) to understand early brain
development better. Model used deep CNN architectures for segmenting
13 × 13 sized input data. Dou et al. [127] and Brosch et al. [128] also used
single path architectures for targeting cerebral microbleed and MS lesions,
respectively. Another strategy of classification is multi-path architectures.
It is an integration of independent operative networks together in the final
model providing more diverse features. Kamnitsas et al. [124] effectively
combined two separate networks in brain lesion segmentation using CNN.
Their network used 3 × 3 × 3 kernels to increase the network depth to
11 layers. They called their complete framework ‘DeepMedic’ [129] which
used eight parallel arranged networks where each network was of seven
layers. Model used this CNN architecture for the anatomy of the complete
brain. Differently, Valverde et al. [130] introduced a cascaded 3D CNN
architecture for MS lesion segmentation using MRI scan images. This can
be classified as an example of a series network. Generally, multi-path archi-
tectures can extract more features and thus lead to better performance.
We have seen examples of 2D and 3D architectures, but Tri-planar
or 2.5D architectures also exist. Like, Lyksborg et al. [131] used a 2.5D
approach to combine processed patches from different planes to pro-
duce desired tumor segmentation results. Today for better 3D modeling
x, y, z cartesian coordinates, and spectral coordinates through eigenvec-
tors are used. After practical segmentation work, some studies also report
post-processing steps to refine the results better. It majorly includes either
creating hard segmentation labels [124] or removing false positives. Some
unsupervised brain image registration works have also gained popularity
in recent years. These require less or almost negligible pre-training experi-
ence; hence it can be an excellent futuristic application. Figure 11.8 depicts
Gaussian noise removal of a standard noise deviation [25] by CNN based
on residual learning i.e., DnCNN in MATLAB [26].
Another application of deep learning here is disease detection.
Alzheimer’s and Parkinson’s disease are common neurodegenerative dis-
eases that affect thousands of people around the globe. In-depth learning-­
based research on Alzheimer’s Disease (AD) is mainly focused on its
detection and classification, which are challenging tasks owing to ambi-
guities in progressive stages and diagnosis criteria of the disease [132].
This age-progressive disease is imaged mainly using MRI or PET-based
imaging modalities. A 2015 study used deep learning-based algorithms
328 Biomedical Data Mining for Information Retrieval

Figure 11.8 Image denoising of brain MR image by DnCNN network [26].

for classification of AD, Mild Cognitive Impairment (MCI), from Control

normal (CN), and MCI converters from non-converter (MCInc) [133].
multi-kernel SVM (MK-SVM) with low-level features (LLF) and SAE
learned features (SAEF) yielded state-of-the-art accuracy in the study.
Hosseini-Asl et al. [134] used a pre-trained 3D CNN from a convolutional
autoencoder (CAE). Model incorporated features from MRI scans. The
purpose here was also to classify AD and MCI from CN. In another study,
Suk and Shen [135] used Deep Ensemble Sparse Regression Network
(DeepESRNet) which is a hybrid of sparse regression models and deep
CNN, to clinically diagnose AD. A prediction based study by Basaia et al.
[136] was to forecast potential AD patients in the upcoming 36 months. The
3D CNN based architecture used here can differentiate AD, MCI patients
from CN and is dataset independent. One recent work of 2019, tries to
enhance performance of AD and MCI detection using sMRI images using
3D densely connected CNNs (3D-DenseNets) [137].
One other neurodegenerative that uses the DL diagnosis approach
is Parkinson’s disease (PD). A lot of deep learning tools have been used
here to analyze the kinematics of the human body for early-stage disease
detection and progression. This is often subdivided as upper [138], and
lower extremities [139] for patients (also called ‘Gait analysis’), which
are recorded by various on or off-body sensors and collected data is then
analyzed using DL based algorithms. DL helps in voice and handwriting
pattern recognition using ANN/SVM and KNN/AdaBoost classifiers. The
DL pipeline for PD diagnosis is quite similar to that of AD diagnosis. In
a 2019 study, Quan et al. [140] use single-photon emission computerized
tomography (SPECT) imaging with Iodine-123 fluoropropyl (123I-FP-
CIT) tracer, resulting in illustrated DaTscan SPECT images. This image
data is then analyzed using DNN based InceptionV3 architecture to effec-
tively classify healthy subjects against diseased ones. Like this, many other
 Deep Learning in Medical Image Analysis 329

classification and detection based studies have been performed in recent

years.

11.4.4.2 Sensory Organs—The Eye and Ear

In ophthalmology, DL is majorly used for the detection of macular degen-
eration, glaucoma, and diabetic retinopathy. The detection of Diabetic
Retinopathy (DR) by a clinician is often delayed and less accurate due to
limited equipment resources and expertise. To aid them, automated tech-
niques like Ref. [141], which uses CNN-based AlexNet/VGG architecture,
were used. System obtained 96.8% sensitivity, 87% specificity, and 0.98
AUG values, proving its precision and accuracy. Another study by Gulshan
et al. [142], screened retinal photographs from adults with diabetes using a
DL model. Model included 128,175 images in their development/training
sets and achieved over 0.99 AUC value over validation/test sets acquired
from EyePACS-1 and Messidor-2 libraries. Some of these DR detection
systems have also been clinically used for years now [143]. AMD or Age-
related Macular Degeneration is an age-onset disease and hence is one
of the primary sources for vision impairment in the elderly population.
Ting et al. [144] used Fovea-centered unsegmented macula images using
DL algorithms to detect developing AMD. Model performed the study on
108,558 retinal images accumulated from 38,189 patients. Grassmann et al.
[145] used the AREDS data set for AMD detection with over 82% sensitiv-
ity using CNN. Even diversified data and reduced overfitting using argu-
mentation tools was used in their system. Another eye-related disorder is
glaucoma, which is often characterized by a defect in optic nerves. Chen
et al. [146] presented a CNN based DL classifier for detecting glaucoma.
Model also attempted to classify retinal images and tested their model on
the Online Retinal Fundus Image dataset for Glaucoma Analysis (ORIGA)
within the 83 to 90% accuracy range. CDR or cup-to-disc ratio was used
here for classification purposes. Much other identification and classifica-
tion based studies are done on glaucoma in recent years, most of which use
CDR as an extracted feature for analysis [147]. While studies like Ref. [148]
used LMeP extracted features using FCM clustering through optimized
cascade correlation neural networks. One new addition by this study was
to add the hidden units one at a time instead of using a predefined set of all
hidden layers, thus minimizing the accrued errors.
Some very recent studies have also reported computer-aided diagno-
sis advancements for middle and external ear conditions through ML/DL
approaches. Like Viscaino et al. [149] utilized SVM, k-nearest neighbor
(k-NN), and decision trees for otoscopic examination. They trained their
330 Biomedical Data Mining for Information Retrieval

model on 700+ ear images. Furthermore, they tested it on 150+ ear images
to characterize the condition of the tympanic membrane and ear canal
with an above 90% classification accuracy and average sensitivity of 87.8%.
As a new crude field, much research is still needed with more in-depth
insights for its better development.

11.4.4.3 Thoracic Cavity

Most imaging modalities for thoracic regions are either based on
Radiography, X-ray, or CT scans. Moreover, many of them are targeted
towards nodules/tumor detection, which indicate developing cancer. We
have already discussed tumor detection and classification extensively using a
segmentation approach in the last (Tissue) section. The primary DL pipeline
used for nodule/tumor analysis generally includes preprocessing (optional),
segmentation, detection, or/and classification using feature extraction and
post-processing (if required). However, this may vary organ to organ.
For lungs, most of the reported advancements are either nodule detec-
tion or classification based. Many recent studies like Ref. [150] use CNN
or DCNN based automated architectures following the same DL pipe-
line, as mentioned above. Huang et al. [151] used CNN for diagnosis of
Interstitial lung disease or ILD (Characterized by abnormal inflammations
of lung tissues). Bermejo-Peláez et al. [152] presented CNN based auto-
mated identification and classification of ILD suspects from CT images,
with state-of-the-art 91.4% sensitivity and 98.1% specificity. Some other
applications include the segmentation of acutely injured lungs and func-
tion quantification.
A similar approach is mostly used for breast cancer diagnosis using
automated tools. The DL pipeline is used extensively here for breast tumor
detection and classification through DCNN and CNN based models like
AlexNet/VGGNet [153]. Ultrasound, Radiography, CT, and MR scans are
some commonly used image modalities here. Low energy X-ray breast scans
are also extensively utilized for imaging breast tissue called Mammography,
which is then analyzed for any abnormalities using DL tools [154]. Some
semi-supervised [155] and weakly supervised [156] learning techniques
for breast cancer diagnosis have been recently developed using DCNN.
Cardiac imaging is both essential and diverse in applications. Motion
analysis, fluid dynamics, electrophysiology, acquisition, and prediction
are typical DL applications in image analysis. A recent study employed
CNN for reconstructing the undersampled MR cardiac images [157].
The proposed model could aggressively reconstruct most of the anatom-
ical structures with high accuracy. Although the study does not provide
 Deep Learning in Medical Image Analysis 331

great insights into pathological aspects of cardiac imaging, instead, it is

more focused on accelerating the data acquisition process. Iterative recon-
struction algorithms in coronary CT angiography are effectively used for
noise and dose reduction during imaging as a preprocessing step. Another
work by Chen et al. [158] aimed at improving the noisy images acquired by
low-dose CT through CNN. Not only 2D/3D images, preferably real-time
medical video denoising using the DL algorithm as a great application to
angiography, is now conclusively demonstrated [159]. Segmentation tech-
niques can be of great aid for cardiovascular physiologists that characterize
heart diseases. Recently reported work by Bai et al. [160] aimed at the seg-
mentation of cardiovascular images by assigning each voxel an anatomical
label using FCN. After training the network on 90,000+ annotated images,
the model proved to be faster and more accurate than manual methods. To
better understand spatial-temporal morphology of heart, Bello et al. [161]
presented an FCN model trained on anatomical shape priors. They used
302 patient imaging data scanned via MRI to predict human survival with
Harrell’s C-index = 0.75 (95% CI: 0.70–0.79) accuracy compared to the
human benchmark of C value of 0.59. Deep learning algorithms are also
effectively used for generating computational fluid dynamic (CFD) models
using segmented images of vessels and creating a detailed topology map of
the vascular plaques and lumen [162]. Some other practical implementa-
tions of DL based cardiac imaging are to identify patient classes with dis-
tinct outcomes in conditions such as heart failure with preserved ejection
fraction (HFpEF) [163]; predicting the risk of future cardiac events using
U-net 3D segments and learned clinically interpretable features from car-
diac motion.

11.4.4.4 Abdomen and Gastrointestinal (GI) Track

Mostly in the abdomen region, segmentation of various organs is a pre-
ferred task in many DL-based studies. Be it segmentation of prostate [164]
or pancreas [165] or kidneys [166] or even bladder, most of them use the
similar earlier described DL pipeline. The main objective of such studies
is to either better understand anatomy and physiology of the respective
organ, or to detect developing tumor/nodules that may lead to malignancy.
Another general application is to detect abnormalities in ducts of various
gut organs. Park et al. [167] is one such great study that extensively reports
developed procedures to annotate GI tract images, including those from
Colon, Duodenum, Gallbladder, both kidneys (L&R), Liver, Pancreas,
Small bowel, Spleen, Stomach and veins. Model used deep network pre-
diction for multi-organ segmentation with 89.4% fidelity acquired from
332 Biomedical Data Mining for Information Retrieval

volumetric CT scans. Some other diverse applications from Colon include

Polyp detection [168]; Colitis detection [169]; Electronic cleansing [170].
Liver lesion [171] and prostate lesion [172] classification are often popu-
larly employed for detecting damaged parts within the organ. The proper
detection and analysis of lesions are significant for internal gut and associ-
ated organs, being almost opaque to the human eye.
Other exciting work by Pei et al. [173] was done to evaluate the frequency
of bowel contractions through a temporal dataset to analyze diameter pat-
terns and streak length. Model used FCN and LSTM (Long short-term
memory) hybrids trained on five cine-MRI unlabeled sequences. Wimmer
et al. [174] and Zhu et al. [175] took advantage of computed endoscopic
imaging modality. Wimmer et al. [174] used it for detection and classifica-
tion of celiac disease using CNN SoftMax in duodenums, while Zhu et al.
[175] utilized automated CNN based feature extraction followed by SVM
for detection and classification of GI lesions.
Most of these advancements are mostly targeted towards early-stage
detection and diagnosis of the gut and associated organ abnormalities. The
futuristic approaches involve constant efforts to develop multi-domain
applicable deep learning tools that can more accurately map most of these
abdominal duct based duct organs in a short time.

11.4.4.5 Other Miscellaneous Applications

There are some other varied miscellaneous applications of DL in medi-
cal image analysis at organ and organ system level. One of them is in
Musculoskeletal applications. This includes identification and segmenta-
tion of various bones, joints, and other soft tissue abnormalities acquired
through imaging modalities like X-Rays, CT, MRI scans, and Ultrasound.
Another great application of DL is associated with the analysis of skin
structure and related diseases. This mainly includes the classification of
skin cancer (or melanoma) and generally follows dermoscopic recogni-
tion. We have covered this miscellaneous section extensively through a
descriptive table below. Table 11.3 also lists some great tools which have
shown utility in multiple domains for image analysis.
The medical image analysis using DL involves the availability of
­application-oriented datasets that can be used for training, testing, and val-
idation of networks. There are various open-access medical image reposito-
ries available for researchers to analyze various medical image modalities like
CT, MRI, ultrasound, and X-rays. In 2018, The National Institutes of Health
(NIH) Clinical Center radiologists released a Deep lesion dataset of 32,000
images to create one unified framework for large scale lesion detection.
 Deep Learning in Medical Image Analysis 333

Table 11.3 Some miscellaneous advancements in the organ section and multi-
domain applicability tool (Inspired from: Litjens et al. [183]).
Source Modality Applications
Dermatology
Yu et al. [176] CNN Lesion segmentation and classification
using Deep residual networks
Esteva et al. [177] CNN Inception network trained 100k+ images
Haenssle et al. CNN Compares CNN performance with
[178] an international group of 50+
dermatologists, Google’s Inception v4
CNN architecture used
Fujisawa et al. CNN Efficient skin tumor classifier, accuracy
[179] 76.5%, sensitivity 96.3%
Brinker et al. [180] CNN Clinical melanoma classification, Par with
dermatologists without clinical training
Muscular-skeletal
Golan et al. [181] CNN CNN (with adversarial component) detects
Hip dysplasia structures and performs
measurements
Jamaludin et al. CNN VGG-19 CNN analyzes vertebral discs;
[182] finds lesion hot spots
Lee et al. [184] CNN Decision supporting system for more
accurate and efficient bone age
assessments (BAAs), >90% accuracy and
<2 s interpretation time
Chaudhari et al. CNN Thin-slice knee MR images from thicker
[186] input slices, 3D CNN, State-of-the-art
Liu et al. [187] CNN Anterior cruciate ligament (ACL) tear
detection within the knee joint, 175
subjects
Li [188] CNN Abnormal sites in radiographic images,
Evaluating musculoskeletal quality
(Continued)
334 Biomedical Data Mining for Information Retrieval

Table 11.3 Some miscellaneous advancements in the organ section and multi-
domain applicability tool (Inspired from: Litjens et al. [183]). (Continued)
Source Modality Applications
Zheng et al. [189] CNN Assessment of children pediatric leg length
discrepancy (LLD), 179 subjects, 96
times faster than humans.
Others
Jaumard-Hakoun RBM Tongue mechanics analysis during the
et al. [190] speech, auto-encoders with RBMs
combination is used
Quinn et al. [191] CNN Smartphone detection of malaria,
tuberculosis & parasite eggs
Lekadir et al. [192] CNN Ultrasound characterization of carotid
plaque composition
Rad et al. [193] CNN Segmentation TE in human blastocyst
image, Synthetic embryo images for 1st
time
Nardelli et al. CNN Convolutional Neural Regressor for airway/
[194] vessel morphology study, Chest CT
Multi-domain deep learning applications
Ghesu et al. [195] CNN Reinforcement learning of head & neck CT
and cardiac MR/US.
Moeskops et al. CNN Segments brain tissues, pectoral muscle &
[196] coronaries using unified network
Roth et al. [197] CNN Multi-stream detection of lymph nodes,
sclerotic lesions, and polyps
Tu et al. [185] CNN Human region-based multi-stream
(HR-MSCNN) model; Full-frame action
recognition

The NIH database contains 100,000 chest X-ray images. The Brainweb is the
most popular simulated and phantom database of MR images available for
public use. Researchers widely use the ‘Mini-mias database’ of mammograms
for breast cancer detection. Some datasets of computer-aided diagnosis com-
petitions organized by Microsoft, Kaggle are also available for the public.
 Deep Learning in Medical Image Analysis 335

11.5 Conclusion and Discussion

The applications of deep learning models in the medical field started as
short experiments in workshops, then appeared in conferences and jour-
nals, and took off since then. Nowadays, DL is omnipresent throughout
the medical community, with CNNs being the most widely used model for
segmentation, detection, and classification tasks. Although there are still
some currently faced challenges by DL methods, let us discuss them along
with possible futuristic solutions.

1. The unavailability of similar format and resolution images

across different sources—Since the equipment used for diag-
nosis varies from place to place, this limits the quality of data
because of current models, which are generally useful only for
single-quality and resolution datasets. One way to deal with
this in the future is to make deep learning more flexible for
input data. Moreover, deep learning algorithms will require
a balanced dataset to learn the underground representations
appropriately, since most of the current medical imaging
dataset is unbalanced, which often leads to class imbalance.
2. The sharing of medical imaging data is much complicated
and challenging today—Due to increased privacy and legal
issues, which also involves global threats to data security.
Solution for this will be more IT (Information technology)
driven by improving the overall scenario of online data
sharing and protection worldwide, addressing critical issues
such as plagiarism along the way.
3. Lack of effective diagnosis for noisy and unclear data via
deep learning—Due to technical challenges like overlap-
ping cells in segmentation data, DL algorithms are yet to be
termed as ‘Completely Robust.’ To avoid this, better optimi-
zation of deep learning models is necessary for the future.
This can be achieved by utilizing combinations of various
training modes and types rather than complete reliance on
the number of learning samples for accuracy training.
4. Issues of authenticity and reliability of DL implementation
in medical diagnosis - By providing negligible explanations
for diagnosis decisions made by a deep learning algorithm,
they are often unable to prove their authenticity and reliabil-
ity. Futuristic solutions for this would include using more
feature costly information extraction and powerful DL tools.
336 Biomedical Data Mining for Information Retrieval

Moreover, improving perception in the future could be

done by bridging the knowledge gap between the existence
of these techniques (In literature and databases) and their
potential users (Medical practitioners or Researches).

This chapter should be seen as one such effort to bridge that gap by
reporting advancements in DL tools and techniques across the medical
image analysis field, in recent years. We have discussed how deep learn-
ing algorithms revolutionized the medical field. We briefly described the
architecture and functioning of different supervised and unsupervised DL
models. We also discussed CNN in some good detail, owing to its popu-
larity and applicability in the field. The advancement section in the chapter
provides a comprehensive and exclusive overview of the reported develop-
ments in medical image analysis in a more systematized biological manner.
The unified aim of it is to motivate the readers, especially medical stake-
holders, to adopt these AI-based networks more effectively in diagnosis.
The challenges listed above can urge one to question the completeness of
DL in the analysis field, but we need to remember that these challenges can
be a matter of the past. This field has exploded in the last decade, and the
pace at which it is improving is just incomparable.
According to statistics, the misdiagnosis rate caused by humans in med-
ical image analysis can reach 10–30% [198]. The sole purpose of DL or
AI-based automated medical diagnosis is to reduce that error percentage
rate. Deep learning or any form of AI can never acquire a human’s place
in medical diagnosis or any other related field since the field is still evolv-
ing and is prone to multiple errors. Furthermore, this evolution itself is a
relative limit which is derived by human’s everlasting ambitions. Thus this
indicates that these AI tools should not be seen as a competitor to medical
doctors or researchers. Instead, they should be better utilized as an aid to
improve accuracy and time efficiency with less tedious procedures to follow.
This can create a new realm of more efficient diagnoses in the future.

References
1. Bradley, W.G., History of Medical Imaging. Proc. Am. Philos. Soc., 152, 3,
349–361, 2008.
2. Willmann, J.K., van Bruggen, N., Dinkelborg, L.M., Gambhir, S.S., Molecular
imaging in drug development. Nat. Rev. Drug Discovery, 7, 7, 591–607, Jul.
2008.
 Deep Learning in Medical Image Analysis 337

3. Hoffman, R.M., Application of GFP imaging in cancer. Lab. Invest., 95, 4,

432–452, Apr. 2015.
4. Grignon, B., Mainard, L., Delion, M., Hodez, C., Oldrini, G., Recent advances
in medical imaging: Anatomical and clinical applications. Surg. Radiol. Anat.,
34, 8. Springer, 675–686, 29-Oct-2012.
5. Li, G. et al., Advances in 4D medical imaging and 4D radiation therapy.
Technol. Cancer Res. Treat., 7, 1, Adenine Press, 67–81, 2008.
6. Canè, F., Verhegghe, B., De Beule, M., Bertrand, P. B., van der Geest, R. J.,
Segers, P., De Santis, G., From 4D Medical Images (CT, MRI, and Ultrasound)
to 4D Structured Mesh Models of the Left Ventricular Endocardium for
Patient-Specific Simulations. BioMed. Res. Int., 7030718, 2018.
7. Lo, S.C.B., Lou, S.L.A., Chien, M.V., Mun, S.K., Artificial Convolution
Neural Network Techniques and Applications for Lung Nodule Detection.
IEEE Trans. Med. Imaging, 14, 4, 711–718, 1995.
8. Brownell, G.L., A History of Positron Imaging. Physics Research Laboratory,
Massachusetts General Hospital, MIT, 1, 1999.
9. Lin, E. and Alessio, A., What are the basic concepts of temporal, contrast,
and spatial resolution in cardiac CT? J. Cardiovasc. Comput. Tomogr., 3, 6,
403–408, Nov. 2009.
10. Kasban, H., El-Bendary, M.A.M., Salama, D.H., A Comparative Study of
Medical Imaging Techniques. Int. J. Inf. Sci. Intell. Syst., 4, 2, 37–38, 2015.
11. Renz, M., Fluorescence microscopy—A historical and technical perspective.
Cytometry Part A, 83, 9, 767–779, Sep. 2013.
12. McCulloch, W.S. and Pitts, W., A logical calculus of the ideas immanent in
nervous activity. Bull. Math. Biophys., 5, 4, 115–133, Dec. 1943.
13. Rosenblatt, F., The perceptron: A probabilistic model for information storage
and organization in the brain. Psychol. Rev., 65, 6, 386–408, 1958.
14. LeCun, Y. et al., Backpropagation Applied to Handwritten Zip Code
Recognition. Neural Comput., 1, 4, 541–551, Dec. 1989.
15. Sharma, N. et al., Automated medical image segmentation techniques. J. Med.
Phys., 35, 1, Wolters Kluwer—Medknow Publications, 3–14, 01-Jan-2010.
16. Hesamian, M.H., Jia, W., He, X., Kennedy, P., Deep Learning Techniques
for Medical Image Segmentation: Achievements and Challenges. J. Digit.
Imaging, 32, 4, 582–596, Aug. 2019.
17. Metz, C.E., Basic principles of ROC analysis. Semin. Nucl. Med., 8, 4, 283–
298, 1978.
18. Hosny, K.M., Kassem, M.A., Foaud, M.M., Classification of skin lesions
using transfer learning and augmentation with Alex-net. PLoS One, 14, 5,
e0217293, 2019.
19. Araújo, T. et al., Classification of breast cancer histology images using
Convolutional Neural Networks. PLoS One, 12, 6, e0177544, Jun. 2017.
20. Kermany, D.S. et al., Identifying Medical Diagnoses and Treatable Diseases
by Image-Based Deep Learning. Cell, 172, 5, 1122–1131.e9, Feb. 2018.
338 Biomedical Data Mining for Information Retrieval

21. Baltruschat, I.M., Nickisch, H., Grass, M., Knopp, T., Saalbach, A.,
Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray
Classification. Sci. Rep., 9, 1, 1–10, Dec. 2019.
22. Diwakar, M. and Kumar, M., A review on CT image noise and its denoising.
Biomed. Signal Process. Control, 42, Elsevier Ltd, 73–88, 01-Apr-2018.
23. Hiremath, P.S., Prema, T., Badiger, S., Speckle Noise Reduction in Medical
Ultrasound Images, in: Advancements and Breakthroughs in Ultrasound
Imaging, InTech, 2013, https://doi.org/10.5772/56519
24. Teymurazyan, A., Riauka, T., Jans, H.S., Robinson, D., Properties of noise in
positron emission tomography images reconstructed with filtered-backpro-
jection and row-action maximum likelihood algorithm. J. Digit. Imaging, 26,
3, 447–456, Jun. 2013.
25. Thakur, R.S., Yadav, R.N., Gupta, L., State-of-art analysis of image denoising
methods using convolutional neural networks. IET Image Process., 13, 13,
2367–2380, 2019.
26. Zhang, K., Zuo, W., Member, S., Chen, Y., Meng, D., Beyond a Gaussian
Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE
Transactions on Image Processing: A Publication of the IEEE Signal Processing
Society, 26, 7, 3142–3155, 2017.
27. Zitova, B., Mathematical approaches for medical image registration, in:
Encyclopedia of Biomedical Engineering, vol. 1–3, pp. 21–32, Elsevier, 2019.
https://doi.org/10.1016/b978-0-12-801238-3.99990-2.
28. Malek, S., Mosleh, M., Dhillon, S.K., Milow, P., Bioimage informatics,
in: Encyclopedia of Bioinformatics and Computational Biology: ABC of
Bioinformatics, vol. 1–3, pp. 993–1010, Elsevier, 2019. https://doi.org/10.
1016/b978-0-12-809633-8.20308-7.
29. Mikolov, T. and Zweig, G., Context dependent recurrent neural network lan-
guage model, in: 2012 IEEE Workshop on Spoken Language Technology, SLT
2012—Proceedings, pp. 234–239, 2012.
30. Graves, A., Mohamed, A., Hinton, G., “Speech Recognition with Deep
Recurrent Neural Networks. ICASSP, IEEE Int. Conf. Acoust. Speech Signal
Process.—Proc, pp. 6645–6649, Mar. 2013.
31. Kalchbrenner, N. and Blunsom, P., Recurrent continuous translation mod-
els, in: EMNLP 2013—2013 Conference on Empirical Methods in Natural
Language Processing, Proceedings of the Conference, 2013.
32. Elman, J.L., Finding structure in time. Cogn. Sci., 14, 2, 179–211, Apr. 1990.
33. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov,
R.R., Improving neural networks by preventing co-adaptation of feature detec-
tors, Jul. 2012. arXiv [cs.NE]. arXiv. http://arxiv.org/abs/1207.0580.
34. Hinton, G.E., Osindero, S., Teh, Y.W., A fast learning algorithm for deep
belief nets. Neural Comput., 18, 7, 1527–1554, May 2006.
35. Bengio, Y., Learning deep architectures for AI. Found. Trends Mach. Learn.,
2, 1, 1–27, 2009.
 Deep Learning in Medical Image Analysis 339

36. Ackley, D.H., Hinton, G.E., Sejnowski, T.J., A learning algorithm for boltz-
mann machines. Cogn. Sci., 9, 1, 147–169, Jan. 1985.
37. Goodfellow, A.C., II and Bengio, Y., Deep Learning. Genet. Program.
Evolvable Mach., 19, 1–2, 305–307, Jun. 2017.
38. Schawinski, K. et al., Generative adversarial networks recover features in
astrophysical images of galaxies beyond the deconvolution limit. MNRAS,
467, 1, L110–L114, 2017.
39. Yang, Q. et al., Low-Dose CT Image Denoising Using a Generative Adversarial
Network With Wasserstein Distance and Perceptual Loss. IEEE Trans. Med.
Imaging, 37, 6, 1348–1357, Jun. 2018.
40. Ker, J., Wang, L., Rao, J., Lim, T., Deep Learning Applications in Medical
Image Analysis. IEEE Access, 6, 9375–9379, Dec. 2017.
41. Hubel, B., Wiesel, A., Hubel, D. N., Wiesel, T. N., Functional Architecture in
the Cat's Visual Cortex from the Neurophysiolojy Laboratory, Department
of Pharmacology Central Nervous System Is the Great Diversity of Its Cell
Types and Inter-Receptive Fields of a More Complex Type (Part I) and to.
Most. 160, 106–154, 1962.
42. Fukushima, K., Neocognitron: A self-organizing neural network model for
a mechanism of pattern recognition unaffected by shift in position. Biol.
Cybern., 36, 4, 193–202, 1980.
43. Rawat, W. and Wang, Z., Deep convolutional neural networks for image
classification: A comprehensive review. Neural Comput., 29, 9, MIT Press
Journals, 2352–2449, 01-Sep-2017.
44. Tivive, F.H.C. and Bouzerdoum, A., Efficient training algorithms for a class
of shunting inhibitory convolutional neural networks. IEEE Trans. Neural
Networks, 16, 3, 541–556, 2005.
45. Mccann, M.T., Jin, K.H., Unser, M., Deep CNN for Inverse Problems in
Imaging. IEEE Signal Process. Mag., 26, 9, 4509–4522, 2017.
46. Jifara, W., Jiang, F., Rho, S., Cheng, M., Liu, S., Medical image denoising using
convolutional neural network: A residual learning approach. J. Supercomput.,
75, 2, 704–718, Feb. 2019.
47. He, K., Zhang, X., Ren, S., Sun, J., Deep residual learning for image recogni-
tion, in: Proceedings of the IEEE Computer Society Conference on Computer
Vision and Pattern Recognition, December, vol. 2016, pp. 770–778, 2016.
48. Ioffe, S. and Szegedy, C., Batch normalization: Accelerating deep network
training by reducing internal covariate shift, in: 32nd International Conference
on Machine Learning, ICML 2015, vol. 1, pp. 448–456, 2015.
49. Jiang, D., Dou, W., Vosters, L., Xu, X., Sun, Y., Tan, T., Denoising of 3D mag-
netic resonance images with multi-channel residual learning of convolu-
tional neural network. Jpn. J. Radiol., 36, 9, 566–574, 2018.
50. Tripathi, P.C. and Bag, S., CNN-DMRI: A Convolutional Neural Network
for Denoising of Magnetic Resonance Images. Pattern Recognit. Lett., 135,
57–63, 2020.
340 Biomedical Data Mining for Information Retrieval

51. Gong, K., Guan, J., Liu, C.-C., Qi, J., PET Image Denoising Using a Deep
Neural Network Through Fine Tuning. IEEE Trans. Radiat. Plasma Med. Sci.,
3, 2, 153–161, 2019.
52. Kadimesetty, V.S., Gutta, S., Ganapathy, S., Yalavarthy, P.K., Convolutional
Neural Network-Based Robust Denoising of Low-Dose Computed
Tomography Perfusion Maps. IEEE Trans. Radiat. Plasma Med. Sci., 3, 2,
137–152, 2019.
53. Pavoni, M., Chang, Y., Park, S., Smedby, O., Convolutional neural net-
work-based image enhancement for x-ray percutaneous coronary interven-
tion. J. Med. Imaging, 5, 02, 1, Jun. 2018.
54. Kumar, K. and Rao, A.C.S., Breast cancer classification of image using con-
volutional neural network, in: Proceedings of the 4th IEEE International
Conference on Recent Advances in Information Technology, RAIT 2018, pp.
1–6, 2018.
55. Diaz-Pinto, A., Morales, S., Naranjo, V., Köhler, T., Mossi, J.M., Navea, A.,
CNNs for automatic glaucoma assessment using fundus images: An exten-
sive validation. Biomed. Eng. Online, 18, 1, 29, Mar. 2019.
56. Carpenter, A.E., Jones, T.R., Lamprecht, M.R., Clarke, C., Kang, I,H., Friman,
O., Guertin, D.A. et al., CellProfiler: Image analysis software for identifying
and quantifying cell phenotypes. Genome Biol., 7, 10, R100, 2006.
57. Handfield, L.F., Strome, B., Chong, Y.T., Moses, A.M., Local statistics allow
quantification of cell-to-cell variability from high-throughput microscope
images. Bioinformatics, 31, 6, 940–947, 2015.
58. Hung, M.-C. and Link, W., Protein localization in disease and therapy. J. Cell
Sci., 124, 20, 3381–3392, Oct. 2011.
59. Boland, M.V. and Murphy, R.F., A neural network classifier capable of recog-
nizing the patterns of all major subcellular structures in fluorescence micro-
scope images of HeLa cells. Bioinformatics, 17, 12, 1213–1223, 2001.
60. Li, J., Newberg, J.Y., Uhlén, M., Lundberg, E., Murphy, R.F., Automated
Analysis and Reannotation of Subcellular Locations in Confocal Images
from the Human Protein Atlas. PLoS One, 7, 11, e50514, Nov. 2012.
61. Tahir, M., Khan, A., Majid, A., Protein subcellular localization of fluores-
cence imagery using spatial and transform domain features. Bioinformatics
28, 1, 91–97, 2012.
62. Yang, F., Liu, Y., Wang, Y., Yin, Z., Yang, Z., MIC_Locator: A novel image-
based protein subcellular location multi-label prediction model based on
multi-scale monogenic signal representation and intensity encoding strat-
egy. BMC Bioinf., 20, 1, 522, Oct. 2019.
63. Pärnamaa, T. and Parts, L., Accurate Classification of Protein Subcellular
Localization from High-Throughput Microscopy Images Using Deep
Learning. G3: Genes|Genomes|Genetics, 7, 5, 1385–1392, May 2017.
64. Kraus, O.Z. et al., Automated analysis of high-content microscopy data with
deep learning. Mol. Syst. Biol., 13, 4, 924, Apr. 2017.
 Deep Learning in Medical Image Analysis 341

65. Scheeder, C., Heigwer, F., Boutros, M., Machine learning and image-based
profiling in drug discovery. Curr. Opin. Syst. Biol., 10, 43–52, Aug. 2018.
66. Ohnuki, S., Oka, S., Nogami, S., Ohya, Y., High-Content, Image-Based
Screening for Drug Targets in Yeast. PLoS One, 5, 4, e10177, Apr. 2010.
67. Wawer, M.J. et al., Toward performance-diverse small-molecule libraries for
cell-based phenotypic screening using multiplexed high-dimensional profil-
ing. Proc. Natl. Acad. Sci. U. S. A., 111, 30, 10911–10916, 2014.
68. Kensert, A., Harrison, P.J., Spjuth, O., Transfer Learning with Deep
Convolutional Neural Networks for Classifying Cellular Morphological
Changes. SLAS Discovery, 24, 4, 466–475, Apr. 2019.
69. Hernández-Mier, Y., Nuño-Maganda, M.A., Polanco-Martagón, S., del R.
García-Chávez, M., Machine Learning Classifiers Evaluation for Automatic
Karyogram Generation from G-Banded Metaphase Images. Appl. Sci., 10, 8,
2758, Apr. 2020.
70. Alexe, G. et al., High expression of lymphocyte-associated genes in node-neg-
ative HER2+ breast cancers correlates with lower recurrence rates. Cancer
Res., 67, 22, 10669–10676, Nov. 2007.
71. Xu, J. et al., Convolutional neural network initialized active contour model
with adaptive ellipse fitting for nuclear segmentation on breast histopatho-
logical images. J. Med. Imaging, 6, 01, 1, Feb. 2019.
72. Sirinukunwattana, K., Raza, S.E.A., Tsang, Y.W., Snead, D.R.J., Cree,
I.A., Rajpoot, N.M., Locality Sensitive Deep Learning for Detection and
Classification of Nuclei in Routine Colon Cancer Histology Images. IEEE
Trans. Med. Imaging, 35, 5, 1196–1206, May 2016.
73. Ounkomol, C., Seshamani, S., Maleckar, M.M., Collman, F., Johnson, G.R.,
Label-free prediction of three-dimensional fluorescence images from trans-
mitted-light microscopy. Nat. Methods, 15, 11, 917–920, Nov. 2018.
74. Liimatainen, K., Valkonen, M., Latonen, L., Ruusuvuori, P., Cell organelle
classification with fully convolutional neural networks, 2018.
75. Su, H., Xing, F., Kong, X., Xie, Y., Zhang, S., Yang, L., Robust cell detection
and segmentation in histopathological images using sparse reconstruction
and stacked denoising autoencoders, in: Lecture Notes in Computer Science
(including subseries Lecture Notes in Artificial Intelligence and Lecture Notes
in Bioinformatics), vol. 9351, pp. 383–390, 2015.
76. Weigert, M. et al., Content-aware image restoration: pushing the limits of
fluorescence microscopy. Nat. Methods, 15, 12, 1090–1097, Dec. 2018.
77. Janowczyk, A., Basavanhally, A., Madabhushi, A., Stain Normalization using
Sparse AutoEncoders (StaNoSA): Application to digital pathology. Comput.
Med. Imaging Graph., 57, 50–61, Apr. 2017.
78. Yang, S.J. et al., Assessing microscope image focus quality with deep learn-
ing. BMC Bioinf., 19, 1, 77, Mar. 2018.
79. Gao, Z., Wang, L., Zhou, L., Zhang, J., HEp-2 Cell Image Classification with
Deep Convolutional Neural Networks. IEEE J. Biomed. Health Inform., 21, 2,
416–428, Mar. 2017.
342 Biomedical Data Mining for Information Retrieval

80. Divya, B.S., Subramaniam, K., Nanjundaswamy, H.R., Human Epithelial

Type-2 Cell Image Classification Using an Artificial Neural Network with
Hybrid Descriptors. IETE J. Res., 66, 1, 30–41, Jan. 2020.
81. Alzubaidi, L., Fadhel, M.A., Al-Shamma, O., Zhang, J., Duan, Y., Deep
Learning Models for Classification of Red Blood Cells in Microscopy Images
to Aid in Sickle Cell Anemia Diagnosis. Electronics, 9, 3, 427, Mar. 2020.
82. Bao, Y., Zhao, X., Wang, L., Qian, W., Sun, J., Morphology-based classifi-
cation of mycobacteria-infected macrophages with convolutional neural
network: Reveal {EsxA-induced} morphologic changes indistinguishable by
naked eyes. Transl. Res., 212, 1–13, Oct. 2019.
83. Wang, S., Yao, J., Xu, Z., Huang, J., Subtype cell detection with an acceler-
ated deep convolution neural network, in: Medical Image Computing and
Computer-Assisted Intervention – MICCAI 2016, pp. 640–648, Springer
International Publishing, 2016.
84. Xie, W., Noble, J.A., Zisserman, A., Microscopy cell counting and detection
with fully convolutional regression networks. Comput. Methods Biomech.
Biomed. Eng. Imaging Vis., 6, 3, 283–292, May 2018.
85. Xu, Z. and Huang, J., Detecting 10,000 cells in one second, in: Medical Image
Computing and Computer-Assisted Intervention – MICCAI 2016, pp. 676–
684, Springer International Publishing, 2016.
86. Chen, T. and Chefd’hotel, C., Deep Learning Based Automatic Immune
Cell Detection for Immunohistochemistry Images, in: Machine Learning in
Medical Imaging, pp. 17–24, 2014.
87. Dong, B., Shao, L., Da Costa, M., Bandmann, O., Frangi, A.F., Deep learning
for automatic cell detection in wide-field microscopy zebrafish images, in:
2015 {IEEE} 12th International Symposium on Biomedical Imaging ({ISBI}),
pp. 772–776, 2015.
88. Zeune, L.L. et al., Deep learning of circulating tumour cells. Nat. Mach.
Intell., 2, 2, 124–133, Feb. 2020.
89. Cireşan, D.C., Giusti, A., Gambardella, L.M., Schmidhuber, J., Mitosis detec-
tion in breast cancer histology images with deep neural networks. Med.
Image Comput. Comput. Assist. Interv., 16, Pt 2, 411–418, 2013.
90. Mahmood, T., Arsalan, M., Owais, M., Lee, M.B., Park, K.R., Artificial
Intelligence-Based Mitosis Detection in Breast Cancer Histopathology
Images Using Faster R-CNN and Deep CNNs. J. Clin. Med. Res., 9, 3, 2020.
https://doi.org/10.3390/jcm9030749.
91. Shkolyar, A., Gefen, A., Benayahu, D., Greenspan, H., Automatic detec-
tion of cell divisions (mitosis) in live-imaging microscopy images using
Convolutional Neural Networks. Conf. Proc. IEEE Eng. Med. Biol. Soc., Aug.
2015, vol. 2015, pp. 743–746.
92. Eulenberg, P. et al., Reconstructing cell cycle and disease progression using
deep learning. Nat. Commun., 8, 1, 463, Sep. 2017.
 Deep Learning in Medical Image Analysis 343

93. Zhong, Q., Busetto, A.G., Fededa, J.P., Buhmann, J.M., Gerlich, D.W.,
Unsupervised modeling of cell morphology dynamics for time-lapse micros-
copy. Nat. Methods, 9, 7, 711–713, May 2012.
94. Chen, Y., Ladi, E., Herzmark, P., Robey, E., Roysam, B., Automated {5-D}
analysis of cell migration and interaction in the thymic cortex from time-
lapse sequences of {3-D} multi-channel multi-photon images. J. Immunol.
Methods, 340, 1, 65–80, Jan. 2009.
95. Pawlowski, N., Caicedo, J.C., Singh, S., Carpenter, A.E., Automating mor-
phological profiling with generic deep convolutional networks. BioRxiv,
2016. https://www.biorxiv.org/content/10.1101/085118v1.abstract.
96. Paul, A., Dey, A., Mukherjee, D.P., Sivaswamy, J., Tourani, V., Regenerative
Random Forest with Automatic Feature Selection to Detect Mitosis in
Histopathological Breast Cancer Images, in: Medical Image Computing and
{Computer-Assisted} Intervention—{MICCAI} 2015, pp. 94–102, 2015.
97. Mao, Y. and Yin, Z., A Hierarchical Convolutional Neural Network for
Mitosis Detection in {Phase-Contrast} Microscopy Images, in: Medical
Image Computing and {Computer-Assisted} Intervention—{MICCAI} 2016,
pp. 685–692, 2016.
98. Balkenhol, M., Karssemeijer, N., Litjens, G.J.S., van der Laak, J., Ciompi, F.,
Tellez, D., {H&E} stain augmentation improves generalization of convolu-
tional networks for histopathological mitosis detection. Medical Imaging
2018: Digital Pathology, 2018.
99. Ouyang, W., Aristov, A., Lelek, M., Hao, X., Zimmer, C., Deep learning mas-
sively accelerates super-resolution localization microscopy. Nat. Biotechnol.,
36, 5, 460–468, Jun. 2018.
100. Li, C., Wang, X., Liu, W., Latecki, L.J., Wang, B., Huang, J., Weakly supervised
mitosis detection in breast histopathology images using concentric loss. Med.
Image Anal., 53, 165–178, Apr. 2019.
101. Rajaraman, S., Jaeger, S., Antani, S.K., Performance evaluation of deep neural
ensembles toward malaria parasite detection in thin-blood smear images.
PeerJ, 7, e6977, May 2019.
102. Shirazi, S.H., Umar, A.I., Naz, S., Razzak, M.I., Efficient leukocyte segmenta­
tion and recognition in peripheral blood image. Technol. Healthcare, 24, 3,
335–347, May 2016.
103. Van Valen, D.A. et al., Deep Learning Automates the Quantitative Analysis
of Individual Cells in {Live-Cell} Imaging Experiments. PloS Comput. Biol.,
12, 11, e1005177, Nov. 2016.
104. Mao, Y., Yin, Z., Schober, J.M., Iteratively training classifiers for circulat-
ing tumor cell detection, in: 2015 {IEEE} 12th International Symposium on
Biomedical Imaging ({ISBI}), pp. 190–194, 2015.
105. Ciompi, F. et al., The importance of stain normalization in colorectal tissue
classification with convolutional networks, in: 2017 {IEEE} 14th International
Symposium on Biomedical Imaging ({ISBI} 2017), pp. 160–163, 2017.
344 Biomedical Data Mining for Information Retrieval

106. Rivenson, Y. et al., Deep Learning Enhanced {Mobile-Phone} Microscopy.

ACS Photonics, 5, 6, 2354–2364, Jun. 2018.
107. Xu, Y., Li, Y., Liu, M., Wang, Y., Lai, M., Chang, E.I.-C., Gland Instance
Segmentation by Deep Multichannel Side Supervision, in: Medical Image
Computing and {Computer-Assisted} Intervention – {MICCAI} 2016, pp. 496–
504, 2016.
108. Shaban, M., Awan, R., Fraz, M.M., Azam, A., Tsang, Y.-W., Snead, D.,
Rajpoot, N.M., Context-Aware Convolutional Neural Network for Grading
of Colorectal Cancer Histology Images. IEEE Trans. Med. Imaging, 39, 7,
2395–2405, 2020.
109. Ehteshami Bejnordi, B. et al., Diagnostic Assessment of Deep Learning
Algorithms for Detection of Lymph Node Metastases in Women With Breast
Cancer. JAMA, 318, 22, 2199–2210, Dec. 2017.
110. Bulten, W. et al., Epithelium segmentation using deep learning in {H&E-
stained} prostate specimens with immunohistochemistry as reference stan-
dard. Sci. Rep., 9, 1, 864, Jan. 2019.
111. Litjens, G. et al., A survey on deep learning in medical image analysis. Med.
Image Anal., 42, 60–88, Dec. 2017.
112. Cruz-Roa, A. et al., High-throughput adaptive sampling for whole-slide his-
topathology image analysis ({HASHI}) via convolutional neural networks:
Application to invasive breast cancer detection. PLoS One, 13, 5, e0196828,
May 2018.
113. Havaei, M. et al., Brain tumor segmentation with Deep Neural Networks.
Med. Image Anal., 35, 18–31, Jan. 2017.
114. Bayramoglu, N., Kannala, J., Heikkilä, J., Deep learning for magnification
independent breast cancer histopathology image classification, in: 2016 23rd
International Conference on Pattern Recognition ({ICPR}), pp. 2440–2445, 2016.
115. Turkki, R. et al., Breast cancer outcome prediction with tumour tissue images
and machine learning. Breast Cancer Res. Treat., 177, 1, 41–52, Aug. 2019.
116. Mobadersany, P. et al., Predicting cancer outcomes from histology and
genomics using convolutional networks. Proc. Natl. Acad. Sci. U. S. A., 115,
13, E2970–E2979, Mar. 2018.
117. Häring, M., Großhans, J., Wolf, F., Eule, S., Automated Segmentation of
Epithelial Tissue Using Cycle-Consistent Generative Adversarial Networks.
https://doi.org/10.1101/311373.
118. Asay, B.C. et al., Digital Image Analysis of Heterogeneous Tuberculosis
Pulmonary Pathology in {Non-Clinical} Animal Models using Deep
Convolutional Neural Networks. Sci. Rep., 10, 1, 6047, Apr. 2020.
119. Bandi, P. et al., From Detection of Individual Metastases to Classification of
Lymph Node Status at the Patient Level: The {CAMELYON17} Challenge.
IEEE Trans. Med. Imaging, 38, 2, 550–560, Feb. 2019.
120. Raza, K. and Singh, N.K., A Tour of Unsupervised Deep Learning for
Medical Image Analysis, Dec. 2018. arXiv [eess.IV]. arXiv. http://arxiv.org/
abs/1812.07715.
 Deep Learning in Medical Image Analysis 345

121. Bernal, J. et al., Deep convolutional neural networks for brain image analysis
on magnetic resonance imaging: A review. Artif. Intell. Med., 95, 64–81, Apr.
2019.
122. Juntu, J., Sijbers, J., Van Dyck, D., Gielen, J., Bias Field Correction for {MRI}
Images, in: Computer Recognition Systems, pp. 543–551, 2005.
123. Pereira, S., Pinto, A., Alves, V., Silva, C.A., Brain Tumor Segmentation Using
Convolutional Neural Networks in {MRI} Images. IEEE Trans. Med. Imaging,
35, 5, 1240–1251, 2016.
124. Kamnitsas, K. et al., Efficient multi-scale {3D} {CNN} with fully connected
{CRF} for accurate brain lesion segmentation. Med. Image Anal., 36, 61–78,
Feb. 2017.
125. Li, W., Wang, G., Fidon, L., Ourselin, S., Cardoso, M.J., Vercauteren, T., On
the Compactness, Efficiency, and Representation of {3D} Convolutional
Networks: Brain Parcellation as a Pretext Task, in: Information Processing in
Medical Imaging, pp. 348–360, 2017.
126. Zhang, W. et al., Deep convolutional neural networks for multi-modality iso-
intense infant brain image segmentation. Neuroimage, 108, 214–224, Mar.
2015.
127. Dou, Q. et al., Automatic Detection of Cerebral Microbleeds From {MR}
Images via {3D} Convolutional Neural Networks. IEEE Trans. Med. Imaging,
35, 5, 1182–1195, May 2016.
128. Brosch, T., Tang, L.Y.W., Yoo, Y., Li, D.K.B., Traboulsee, A., Tam, R., Deep
{3D} Convolutional Encoder Networks With Shortcuts for Multiscale
Feature Integration Applied to Multiple Sclerosis Lesion Segmentation. IEEE
Trans. Med. Imaging, 35, 5, 1229–1239, May 2016.
129. de Brebisson, A. and Montana, G., Deep neural networks for anatomical
brain segmentation, in: Proceedings of the {IEEE} Conference on Computer
Vision and Pattern Recognition Workshops, pp. 20–28, 2015.
130. Valverde, S. et al., Improving automated multiple sclerosis lesion seg-
mentation with a cascaded {3D} convolutional neural network approach.
Neuroimage, 155, 159–168, Jul. 2017.
131. Lyksborg, M., Puonti, O., Agn, M., Larsen, R., An Ensemble of {2D}
Convolutional Neural Networks for Tumor Segmentation, in: Image Analysis,
pp. 201–211, 2015.
132. Ebrahimighahnavieh, M.A., Luo, S., Chiong, R., Deep learning to detect
Alzheimer’s disease from neuroimaging: A systematic literature review.
Comput. Methods Programs Biomed., 187, 105242, Apr. 2020.
133. Suk, H.-I., Lee, S.-W., Shen, D., Alzheimer’s Disease Neuroimaging Initiative,
Latent feature representation with stacked auto-encoder for {AD/MCI} diag-
nosis. Brain Struct. Funct., 220, 2, 841–859, Mar. 2015.
134. Hosseini-Asl, E., Gimel, F.G., El-Baz, A., Alzheimer’s Disease Diagnostics
by Adaptation of 3D Convolutional Network. In 2016 IEEE International
Conference on Image Processing (ICIP), 2016. https://doi.org/10.1109/
icip.2016.7532332.
346 Biomedical Data Mining for Information Retrieval

135. Suk, H.-I. and Shen, D., Deep ensemble sparse regression network for
Alzheimer’s disease diagnosis, in: International Workshop on Machine
Learning in Medical Imaging, pp. 113–121, 2016.
136. Basaia, S. et al., Automated classification of Alzheimer’s disease and mild
cognitive impairment using a single {MRI} and deep neural networks.
NeuroImage Clin., 21, 101645, 2019.
137. Wang, H. et al., Ensemble of {3D} densely connected convolutional net-
work for diagnosis of mild cognitive impairment and Alzheimer’s disease.
Neurocomputing, 333, 145–156, 2019.
138. Fraiwan, L., Khnouf, R., Mashagbeh, A.R., Parkinson’s disease hand tremor
detection system for mobile application. J. Med. Eng. Technol., 40, 3, 127–
134, 2016.
139. Jane, Y.N., Nancy Jane, Y., Khanna Nehemiah, H., Arputharaj, K., A
Q-backpropagated time delay neural network for diagnosing severity of gait
disturbances in Parkinson’s disease. J. Biomed. Inf., 60, 169–176, 2016.
140. Quan, J., Xu, L., Xu, R., Tong, T., Su, J., DaTscan SPECT Image Classification
for Parkinson’s Disease, 2019. arXiv [eess.IV]. arXiv. http://arxiv.org/
abs/1909.04142.
141. Abràmoff, M.D. et al., Improved Automated Detection of Diabetic
Retinopathy on a Publicly Available Dataset Through Integration of Deep
Learning. Invest. Ophthalmol. Vis. Sci., 57, 13, 5200–5206, Oct. 2016.
142. Gulshan, V. et al., Development and Validation of a Deep Learning Algorithm
for Detection of Diabetic Retinopathy in Retinal Fundus Photographs.
JAMA, 316, 22, 2402–2410, Dec. 2016.
143. Abràmoff, M.D., Lavin, P.T., Birch, M., Shah, N., Folk, J.C., Pivotal trial of an
autonomous {AI-based} diagnostic system for detection of diabetic retinop-
athy in primary care offices. NPJ Digit Med., 1, 39, Aug. 2018.
144. Ting, D.S.W. et al., Development and Validation of a Deep Learning System
for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From
Multiethnic Populations With Diabetes. JAMA, 318, 22, 2211–2223, 2017.
145. Grassmann, F. et al., A Deep Learning Algorithm for Prediction of {Age-
Related} Eye Disease Study Severity Scale for {Age-Related} Macular
Degeneration from Color Fundus Photography. Ophthalmology, 125, 9,
1410–1420, Sep. 2018.
146. Chen, X., Xu, Y., Yan, S., Wong, D.W.K., Wong, T.Y., Liu, J., Automatic Feature
Learning for Glaucoma Detection Based on Deep Learning, in: Medical
Image Computing and {Computer-Assisted} Intervention—{MICCAI} 2015,
pp. 669–677, 2015.
147. Puthren, P.R., Agrawal, A., Padma, U., Automated Glaucoma Detection
Using Global Statistical Parameters of Retina Fundus Images, in: Proceedings
of the International Conference on {ISMAC} in Computational Vision and
{Bio-Engineering} 2018 ({ISMAC-CVB}), pp. 377–388, 2019.
 Deep Learning in Medical Image Analysis 347

148. Devasia, T., Jacob, K.P., Thomas, T., Automatic Early Stage Glaucoma
Detection Using Cascade Correlation Neural Network, in: Smart Intelligent
Computing and Applications, pp. 659–669, 2019.
149. Viscaino, M., Maass, J.C., Delano, P.H., Torrente, M., Stott, C., Auat Cheein,
F., Computer-aided diagnosis of external and middle ear conditions: A
machine learning approach. PLoS One, 15, 3, e0229226, Mar. 2020.
150. Nishio, M. et al., Computer-aided diagnosis of lung nodule classification
between benign nodule, primary lung cancer, and metastatic lung cancer at
different image size using deep convolutional neural network with transfer
learning. PLoS One, 13, 7, e0200721, Jul. 2018.
151. Huang, S., Lee, F., Miao, R., Si, Q., Lu, C., Chen, Q., A deep convolutional
neural network architecture for interstitial lung disease pattern classification.
Med. Biol. Eng. Comput., 58, 4, 725–737, Apr. 2020.
152. Bermejo-Peláez, D., Ash, S.Y., Washko, G.R., San José Estépar, R., Ledesma-
Carbayo, M.J., Classification of Interstitial Lung Abnormality Patterns with
an Ensemble of Deep Convolutional Neural Networks. Sci. Rep., 10, 1, 338,
Jan. 2020.
153. Chiang, T.-C., Huang, Y.-S., Chen, R.-T., Huang, C.-S., Chang, R.-F., Tumor
Detection in Automated Breast Ultrasound Using {3-D} {CNN} and Prioritized
Candidate Aggregation. IEEE Trans. Med. Imaging, 38, 1, 240–249, Jan. 2019.
154. Sahiner, B. et al., Classification of mass and normal breast tissue: a convolu-
tion neural network classifier with spatial domain and texture images. IEEE
Trans. Med. Imaging, 15, 5, 598–610, 1996.
155. Sun, W., Tseng, T.-L.B., Zhang, J., Qian, W., Enhancing deep convolutional
neural network scheme for breast cancer diagnosis with unlabeled data.
Comput. Med. Imaging Graph., 57, 4–9, Apr. 2017.
156. Hwang, S. and Kim, H.-E., {Self-Transfer} Learning for Weakly Supervised
Lesion Localization, in: Medical Image Computing and {Computer-Assisted}
Intervention—{MICCAI} 2016, pp. 239–246, 2016.
157. Schlemper, J., Caballero, J., Hajnal, J.V., Price, A.N., Rueckert, D., A Deep
Cascade of Convolutional Neural Networks for Dynamic {MR} Image
Reconstruction. IEEE Trans. Med. Imaging, 37, 2, 491–503, Feb. 2018.
158. Chen, H. et al., Low-dose {CT} via convolutional neural network. Biomed.
Opt. Express, 8, 2, 679–694, Feb. 2017.
159. Sadda, P. and Qarni, T., {Real-Time} Medical Video Denoising with Deep
Learning: Application to Angiography. Int. J. Appl. Inf. Syst., 12, 13, 22–28,
May 2018.
160. Bai, W. et al., Automated cardiovascular magnetic resonance image analysis
with fully convolutional networks. J. Cardiovasc. Magn. Reson., 20, 1, 65, Sep.
2018.
161. Bello, G.A. et al., Deep learning cardiac motion analysis for human survival
prediction. Nat. Mach. Intell., 1, 95–104, Feb. 2019.
162. Taylor, C.A., Fonte, T.A., Min, J.K., Computational fluid dynamics applied to
cardiac computed tomography for noninvasive quantification of fractional
348 Biomedical Data Mining for Information Retrieval

flow reserve: Scientific basis. J. Am. Coll. Cardiol., 61, 22, 2233–2241, Jun.
2013.
163. Shah, S.J. et al., Phenomapping for novel classification of heart failure with
preserved ejection fraction. Circulation, 131, 3, 269–279, Jan. 2015.
164. Orlando, N., Gillies, D.J., Gyacskov, I., Romagnoli, C., D’Souza, D., Fenster,
A., Automatic prostate segmentation using deep learning on clinically
diverse {3D} transrectal ultrasound images. Med. Phys., Mar. 2020. https://
doi.org/10.1002/mp.14134.
165. Roth, H.R., Farag, A., Lu, L., Turkbey, E.B., Summers, R.M., Deep convo-
lutional networks for pancreas segmentation in {CT} imaging, in: Medical
Imaging 2015: Image Processing, vol. 9413, p. 94131G, 2015.
166. Thong, W., Kadoury, S., Piché, N., Pal, C.J., Convolutional networks for
kidney segmentation in contrast-enhanced {CT} scans. Comput. Methods
Biomech. Biomed. Eng. Imaging Vis., 6, 3, 277–282, May 2018.
167. Park, S. et al., Annotated normal {CT} data of the abdomen for deep learn-
ing: Challenges and strategies for implementation. Diagn. Interv. Imaging,
101, 1, 35–44, Jan. 2020.
168. Zhang, R. et al., Automatic Detection and Classification of Colorectal Polyps
by Transferring {Low-Level} {CNN} Features From Nonmedical Domain.
IEEE J. Biomed. Health Inform., 21, 1, 41–47, Jan. 2017.
169. Liu, J. et al., Colitis detection on computed tomography using regional con-
volutional neural networks, in: 2016 {IEEE} 13th International Symposium on
Biomedical Imaging ({ISBI}), pp. 863–866, 2016.
170. Tachibana, R., Näppi, J.J., Hironakaa, T., Kim, S.H., Yoshida, H., Deep learn-
ing for electronic cleansing in dual-energy {CT} colonography, in: Medical
Imaging 2016: {Computer-Aided} Diagnosis, vol. 9785, p. 97851M, 2016.
171. Vivanti, R., Ephrat, A., Joskowicz, L., Karaaslan, O., Lev-Cohain, N., Sosna,
J., Automatic liver tumor segmentation in follow-up {CT} studies using
convolutional neural networks, in: Proc. {Patch-Based} Methods in Medical
Image Processing Workshop, vol. 2, 2015.
172. Zhu, Y. et al., {MRI-based} prostate cancer detection with high-level repre-
sentation and hierarchical classification. Med. Phys., 44, 3, 1028–1039, Mar.
2017.
173. Pei, M., Wu, X., Guo, Y., Fujita, H., Small bowel motility assessment based
on fully convolutional networks and long short-term memory. Knowl.-Based
Syst., 121, 163–172, Apr. 2017.
174. Wimmer, G., Vécsei, A., Uhl, A., {CNN} transfer learning for the automated
diagnosis of celiac disease, in: 2016 Sixth International Conference on Image
Processing Theory, Tools and Applications ({IPTA}), pp. 1–6, 2016.
175. Zhu, R., Zhang, R., Xue, D., Lesion detection of endoscopy images based on
convolutional neural network features, in: 2015 8th International Congress on
Image and Signal Processing ({CISP}), pp. 372–376, 2015.
 Deep Learning in Medical Image Analysis 349

176. Yu, L., Chen, H., Dou, Q., Qin, J., Heng, P.-A., Automated Melanoma
Recognition in Dermoscopy Images via Very Deep Residual Networks. IEEE
Trans. Med. Imaging, 36, 4, 994–1004, Apr. 2017.
177. Esteva, A. et al., Dermatologist-level classification of skin cancer with deep
neural networks. Nature, 542, 7639, 115–118, Feb. 2017.
178. Haenssle, H.A. et al., Man against machine: Diagnostic performance of a deep
learning convolutional neural network for dermoscopic melanoma recognition
in comparison to 58 dermatologists. Ann. Oncol., 29, 8, 1836–1842, Aug. 2018.
179. Fujisawa, Y. et al., Deep-learning-based, computer-aided classifier developed
with a small dataset of clinical images surpasses board-certified dermatolo-
gists in skin tumour diagnosis. Br. J. Dermatol., 180, 2, 373–381, Feb. 2019.
180. Brinker, T.J. et al., A convolutional neural network trained with dermoscopic
images performed on par with 145 dermatologists in a clinical melanoma
image classification task. Eur. J. Cancer, 111, 148–154, Apr. 2019.
181. Golan, D., Donner, Y., Mansi, C., Jaremko, J., Ramachandran, M., Others,
Fully automating Graf ’s method for {DDH} diagnosis using deep convo-
lutional neural networks, in: Deep Learning and Data Labeling for Medical
Applications, 2016. https://doi.org/10.1007/978-3-319-46976-8_14.
182. Jamaludin, A., Kadir, T., Zisserman, A., {SpineNet}: Automatically
Pinpointing Classification Evidence in Spinal {MRIs}, in: Medical Image
Computing and {Computer-Assisted} Intervention—{MICCAI} 2016, pp. 166–
175, 2016.
183. Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian,
M. et al., A survey on deep learning in medical image analysis. Med. Image
Anal., 42, 60–88, 2017.
184. Lee, H. et al., Fully Automated Deep Learning System for Bone Age
Assessment. J. Digit. Imaging, 30, 4, 427–441, Aug. 2017.
185. Tu, Z., Xie, W., Qin, Q., Poppe, R., Veltkamp, R.C., Li, B. et al., Multi-stream
CNN: Learning representations based on human-related regions for action
recognition. Pattern Recognit., 79, 32–43, 2018.
186. Chaudhari, A.S. et al., Super-resolution musculoskeletal {MRI} using deep
learning. Magn. Reson. Med., 80, 5, 2139–2154, Nov. 2018.
187. Liu, F. et al., Fully Automated Diagnosis of Anterior Cruciate Ligament Tears
on Knee {MR} Images by Using Deep Learning. Radiol. Artif. Intell., 1, 3,
180091, May 2019.
188. Li, S., Deep adversarial model for musculoskeletal quality evaluation. Inf.
Process. Manage., 57, 1, 102146, Jan. 2020.
189. Zheng, Q., Shellikeri, S., Huang, H., Hwang, M., Sze, R.W., Deep Learning
Measurement of Leg Length Discrepancy in Children Based on Radiographs.
Radiology, 2020. https://doi.org/10.1148/radiol.2020192003.
190. Jaumard-Hakoun, A., Xu, K., Roussel-Ragot, P., Dreyfus, G., Denby, B.,
Tongue contour extraction from ultrasound images based on deep neural net-
work, May 2016. arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1605.05912.
 350 Biomedical Data Mining for Information Retrieval

191. Quinn, J.A., Nakasi, R., Mugagga, P.K.B., Byanyima, P., Lubega, W., Andama,
A., Deep Convolutional Neural Networks for {Microscopy-Based} Point of Care
Diagnostics, in: Machine Learning for Healthcare Conference, pp. 271–281, 2016.
192. Lekadir, K. et al., A Convolutional Neural Network for Automatic
Characterization of Plaque Composition in Carotid Ultrasound. IEEE J.
Biomed. Health Inf., 21, 1, 48–55, Jan. 2017.
193. Rad, R.M., Saeedi, P., Au, J., Havelock, J., Trophectoderm segmentation
in human embryo images via inceptioned {U-Net}. Med. Image Anal., 62,
101612, Feb. 2020.
194. Nardelli, P., Ross, J.C., San José Estépar, R., Generative-based airway and ves-
sel morphology quantification on chest {CT} images. Med. Image Anal., 63,
101691, Mar. 2020.
195. Ghesu, F.C., Georgescu, B., Mansi, T., Neumann, D., Hornegger, J.,
Comaniciu, D., An Artificial Agent for Anatomical Landmark Detection
in Medical Images, in: Medical Image Computing and {Computer-Assisted}
Intervention—{MICCAI} 2016, pp. 229–237, 2016.
196. Moeskops, P. et al., Deep Learning for Multi-task Medical Image
Segmentation in Multiple Modalities, in: Medical Image Computing and
{Computer-Assisted} Intervention—{MICCAI} 2016, pp. 478–486, 2016.
197. Roth, H.R. et al., Improving {Computer-Aided} Detection Using
Convolutional Neural Networks and Random View Aggregation. IEEE
Trans. Med. Imaging, 35, 5, 1170–1181, May 2016.
198. Kerlikowske, K. et al., Performance of screening mammography among
women with and without a first-degree relative with breast cancer. Ann.
Intern. Med., 133, 11, 855–863, Dec. 2000.

View publication stats