Abstract

This paper proposes a novel medical image fusion algorithm that enhances image
quality and preserves input details. Using Gaussian and rolling guidance filters,
images are decomposed into five components: two high-detail (HDCs), two low-
detail (LDCs), and a base component (BC). A modified VGG19 (MVGG-19)
network, adapted via transfer learning, is employed for classifying input modalities
(MRI, CT, PET, SPECT) and establishing fusion rules for HDCs and LDCs. For
BCs, a coupled neural P system (CNPS) defines fusion rules. Experimental results
demonstrate superior image quality and effective information retention compared
to seven advanced algorithms.

1.Introduction

Rising Importance of Medical Image Synthesis

 Medical image synthesis has become increasingly important for improving

clinical diagnosis.
 Fusion of different medical imaging modalities (e.g., MRI, CT, PET,
SPECT) combines their strengths:
o MRI: High soft-tissue resolution.
o CT: Better at showing dense tissue.
o PET and SPECT: Provide metabolic information but have low
spatial resolution.

Image Synthesis Process

 Comprises three main steps:

1. Image decomposition: Extracts complementary information layers.
2. Component synthesis: Merges components using specific rules.
3. Image reconstruction: Converts synthesized components back to the
spatial domain.

Image Decomposition Methods

 Various techniques used, each with strengths and limitations:

 o Laplacian pyramid: Multiscale representation of frequency
information.
o NSCT: Handles intricate textures and edges.
o NSST: Effective for analyzing complex structures.
o Sparse Representation (SR): Linear combination of dictionary
elements.
o Two-scale decomposition: Separates into low-frequency base and
high-frequency detail layers.
o L1-L0 decomposition: Uses regularization norms for layer
separation.

Synthesis Rules for Components

 Low-frequency component rules:

o Averaging and max selection are commonly used but may reduce
quality or detail.
o Adaptive fusion rules ensure image quality.
 High-frequency component rules: Include advanced neural techniques and
optimization schemes.

Advances in Deep Learning for Image Fusion

 Two main groups of deep learning models:

1. Networks for synthesizing specific image components.
2. Closed-loop networks for full image synthesis.
 Examples: M4Net, DEFNet, and MSENet.

Meta-heuristic Optimization in Image Fusion

 Optimization algorithms improve fusion quality.

 Techniques include PSO, CSA, and STV for adaptive rule generation.
 These methods are effective but computationally intensive.

Limitations of Current Methods

 Decomposition methods (e.g., LP, NSCT, SR) have specific shortcomings,

like information loss or color distortion.
 High-frequency synthesis methods lack efficiency.
 Pre-trained deep models struggle with diverse medical modalities.
 Some low-frequency fusion rules (e.g., averaging) degrade image quality.
Proposed Methodology for Overcoming Limitations

 Introduces a novel approach addressing image quality and detail

preservation:
o Image decomposition (GF-RGF): Uses Gaussian and rolling
guidance filters for five components (HDCs, LDCs, BC).
o Fusion rules (HDCs and LDCs): Employs MVGG-19 with enhanced
features via local energy and STS operators.
o CNPS-based fusion for low-frequency components: Ensures robust
image quality against noise.
o Demonstrates high efficacy in preserving input details and improving
composite image quality.

Contributions of the Study

 Proposed decomposition and fusion methods improve medical image

synthesis.
 Introduction of MVGG-19 and CNPS for better feature representation and
noise robustness.
 Ensures the resulting images retain input information with enhanced quality.

Structure of the Paper

 Section 2: Background on key techniques (RGF, structure tensor, CNPS).

 Section 3: Details the proposed methods (decomposition, feature
enrichment, fusion rules).
 Section 4: Experimental datasets, metrics, and results.
 Section 5: Summary and future research directions.

2.Background

2.1, Rolling guidance filter (RGF)

- Step 1: Eliminate small structures (Gaussian filter – a smooth filter)  improving

the overall quality and clarity of medical images.
- Step 2: Restore the edge  preserve edges and important structures within the
medical images  it can effectively enhance edges and fine details in the images
without compromising their sharpness.

2.2. Structure tensor and structure tensor saliency (STS)

- Gradient Features (1st order directional derivative) is able to accurately reflect the
important perceptual textures, edges, and geometrical structures of organs, tissues,
and fibers.

- The structure tensor is an effective tool for analyzing local gradient features.

2.3 Coupled neural P system (CNPS)

- Using for image segmentation, image denoising, image fusion in medical imaging

- Advantage of CNPS: Handling large datasets, flexibility to adapt and extend the
system, handling high-resolution medical images, integration of neural processing
enables better detection and segmentation outcomes

3. The proposed algorithms

3.1 The GF-RGF image decomposition method

- Image decomposition plays a vital role in image fusion but 2-component

decomposition methods such as average or low-pass filter often damage image
details.

- Combines Gaussian filter (GF) and rolling guidance filter (RGF) to decompose
an image into five components: one base, two highly detailed, and two low-
detailed components.
- Preserves edges and fine details while reducing noise, suitable for tasks like
medical image fusion, object recognition, and denoising and offers computational
efficiency and adaptability for real-time applications.

- Edge-preserving filtering takes advantage of full spatial information, providing

diverse analysis results compared to multi-scale transformation methods.

3.2 A feature enrichment method based on the LE function and STS operator

- MVGG-19 Network Features: The features extracted from the MVGG-19 network (pool1 to
pool5) are of lower quality and appear indistinct or blurred, making them challenging to analyze
effectively using the STS operator.

- STS Operator Limitation: The STS operator is ineffective at detecting faint or small features
in the MVGG-19 extracted features.

- Proposed Solution: A new feature enrichment method is introduced by combining the local
energy function with the STS operator to improve feature detection.

- Local Energy Function: The local energy (LE) is calculated using a sliding window over the
high-frequency components of the image, summing the squares of coefficient values within the
window.

as the Hadamard product, denoted as LE_STS = S(I) ∘ LE(I), where S is the STS operator.
- Mathematical Integration: The integration of local energy with the STS operator is expressed

- Improved Detection: The LE_STS method significantly enhances the detection of weak and
minute features, as seen in the comparison of STS(Wi) and LE_STS(Wi) feature images.

3.3 Fusion rules based on the modified VGG-19 (FR_MVGG_19)

1. Pre-trained Networks for Image Synthesis:

o Pre-trained models often struggle with medical images from different modalities,
leading to loss of detail.
2. Proposed Solution: Transfer Learning with VGG-19:
o To address the limitations of pre-trained models, the authors propose using
transfer learning with the VGG-19 network.
o VGG-19 was selected for its deep architecture and ability to capture fine-grained
features due to its use of small convolutional filters (3x3) in multiple layers.
3. Network Architecture and Image Synthesis:
o The MVGG-19 network was trained using 1424 images from MRI, CT, PET, and
SPECT modalities.
 o The network was designed to synthesize high-frequency components in medical
images (referred to as FR_MVGG_19).
4. Steps for Image Synthesis:
o Step 1: Generate 3-D images (LMRI and LSPECT) from high-frequency
components L1 and L2.
o Step 2: Extract features from six layers (input, pool1 to pool5) of the MVGG-19
network for the two input components.
o Step 3: Perform averaging to obtain 2D features for each layer.
o Step 4: Use bilinear interpolation for upsampling the features to 256x256 size.
o Step 5: Enrich the features using the LE_STS method and the local energy
function.
o Step 6: Define decision maps (DMi) for each feature set, and combine them to
create a final decision map (FDM).
o Step 7: Use the FDM to combine the high-frequency components L1 and L2 to
synthesize the final component (LF).
5. Final Outcome:
o The composite component (LF) is generated by combining the decision maps and
the high-frequency components.

This method helps enhance medical image synthesis by improving the representation of high-
frequency features.

3.4 Fusion rules based on CNPSs (FR_CNPS)

 Coupled Neural P Systems (CNPSs):

 CNPSs are Turing-universal, distributed parallel computing models.

 CNPSs have been used for image fusion, with applications in fusing different
components of images.
o Wang et al. [57] used CNPSs for fusing high-detail components.
o Li et al. [56] suggested CNPSs for fusing low-frequency components.
 Limitations in Fusing Base Components:
o The average rule, often used for base component fusion, has low computational
complexity but may cause a decrease in brightness and contrast in the fused
image.

 Proposed Approach (FR_CNPS):

 The proposed method, FR_CNPS, uses CNPSs to fuse base components while
maintaining brightness and contrast.
 CNPS1 and CNPS2 are two CNPSs with local topology that take the base components
of input images (I1, I2) as external inputs.
 Excitation Matrices (C1 and C2):
o C1 and C2 represent the number of times neurons fire in CNPS1 and CNPS2.
 Fusion Rule for Base Components: --- in paper ---
 3.5 The proposed algorithms

 Input Transformation: The input color image (IP) is converted into the YUV color model.
 Decomposition (GF-RGR Algorithm): The image (IM) and the Y-channel are decomposed
into:

 Highly-detailed components (HDCs)

 Low-detailed components (LDCs)
 Base components (BCs)

 Fusion (FR_MVGG_19 and FR_CNPS methods):

 The HDCs and LDCs are fused using the FR_MVGG_19 method.
 The BCs are fused using the FR_CNPS method.

 Combination: The fused components (HDCs, LDCs, BCs) are combined to generate the
fused image, FGray.

 Final Transformation: The FGray image is transformed back into the RGB color space
using the FGray, U, and V channels.

4 Experiments and results

4.1 Evaluation indexes

We select eight commonly used metrics.

• Contrast index (CI).
• Average gradient (AG).
• Mutual information (MI).
• Nonlinear correlation information entropy (NCIE).
• TMQI (Tone Mapped image Quality Index).
• Gradient-based fusion performance QAB=F.

• Qw and Qe (Peilla’s metric).

4.2 Experimental setup

4.2.1 Datasets
- 64 pairs of MRI and SPECT images from Groups 1, 2, and 3 in Table 1  used
to evaluate the proposed image synthesis method.

- Group 4 contains 1424 images from four different types of imaging techniques
(MRI, CT, PET, and SPECT), which were utilized to construct the MVGG-19
model.

4.2.2 Comparison algorithms

4.2.3 Training details

4.3 Experimental results and Evaluation

 Visual Evaluation:

 The proposed method produces superior image quality in terms of brightness, contrast,
and sharpness compared to other fusion algorithms.
 Small frame extractions show that the proposed method generates sharper images,
preserving both soft tissue (functional) and structural (skull) information.
 Color preservation is superior in the proposed method, whereas other fusion methods
result in color degradation or distortion.

 Quantitative Evaluation:

 Group 1 (Image Quality Metrics):

o CI, AG, and TMQI metrics show that the proposed method outperforms others in
contrast and sharpness. The CI value of 0.2499 is the highest among the compared
methods.
 Group 2 (Information Theory Metrics):
o MI and NCIE metrics show that the proposed method retains more information
and incurs less information loss compared to other models. The NCIE index of
0.8161 is the highest.
 Group 3 (Structural & Feature Similarity Metrics):
o QAB=F, Qw, and Qe metrics reveal that the proposed method preserves structural
and feature information better than other methods. The QAB=F value of 0.8020 is
notably higher.

 Conclusion:

 The proposed method excels in both image quality and information preservation,
outperforming other image fusion methods across all evaluation metrics.

4.4 Computation cost

 Computation Cost Overview: The computation cost is used to evaluate algorithm efficiency.
 Algorithm A1:

 Highest running time among all algorithms.

 Times: 18.4934s (Group 1), 18.2735s (Group 2), 18.4263s (Group 3).
 Uses the NSCT transform method, which increases decomposition time.

 Algorithm A3:

 Quickest execution time.

 Times: 0.0994s (Group 1), 0.1066s (Group 2), 0.1065s (Group 3).

 Proposed Method:

 Execution times: 2.0538s (Group 1), 2.1575s (Group 2), 2.1221s (Group 3).
 Ranked 4th in computational cost compared to seven other algorithms.

5 Conclusion
 Study Focus:

 The study introduces a new approach to synthesizing medical images, focusing on

balancing image quality and information retention.
 Three new algorithms are proposed for improving image synthesis efficiency and quality.

 Proposed Algorithms:

 Algorithm 1 (Image Decomposition): Divides the input image into five components:
two high-detail components (HDCs), two low-detail components (LDCs), and a base
component (BC). This decomposition enhances synthesis efficiency.
 Algorithm 2 (Fusion Method for HDCs and LDCs): Based on the FR_MVGG_19
method, this algorithm preserves fine details. A feature enrichment technique, combining
local energy function and STS operator, is introduced to improve this method’s
efficiency.
 Algorithm 3 (Fusion Rule for BC): Based on CNPS, this fusion rule prevents image
quality degradation during synthesis.

 Experimental Evaluation:

 Sixty-four pairs of MRI-SPECT images were used for testing.

 Seven state-of-the-art fusion methods and eight performance indicators were evaluated.
 The results show that the proposed method outperforms other methods in both image
quality and information preservation.
 Future Improvements:

 Enhancing Input Image Quality: High-quality input images improve synthesis

efficiency and better represent image features.
 Improving FR_CNPS Rule: Optimization algorithms based on meta-heuristics can
improve the efficiency of the FR_CNPS fusion rule, allowing optimal control over the
quality of the output image.

 Execution Time:

 The proposed method had a mean execution time (MET) of around 2.05-2.16 seconds
across different datasets, showing efficiency in comparison to other methods.