A Tiered GAN Approach for Monet-Style Image

Generation
FNU Neha Deepshikha Bhati Deepak Kumar Shukla Md Amiruzzaman
Dept. of Computer Science Dept. of Computer Science Rutgers Business School Dept. of Computer Science
Kent State University Kent State University Rutgers University West Chester University
Kent, OH, USA Kent, OH, USA Newark, New Jersey, USA West Chester, PA, USA
[email protected] [email protected] [email protected] [email protected]
arXiv:2412.05724v1 [cs.CV] 7 Dec 2024

Abstract—Generative Adversarial Networks (GANs) have

proven to be a powerful tool in generating artistic images,
capable of mimicking the styles of renowned painters, such
as Claude Monet. This paper introduces a tiered GAN model
to progressively refine image quality through a multi-stage
process, enhancing the generated images at each step. The model
transforms random noise into detailed artistic representations,
addressing common challenges such as instability in training,
mode collapse, and output quality. This approach combines Fig. 1: An overview of how GAN model works
downsampling and convolutional techniques, enabling the gen-
eration of high-quality Monet-style artwork while optimizing Building on the success of GANs in generating realistic and
computational efficiency. Experimental results demonstrate the artistic imagery, this work leverages their capabilities to create
architecture’s ability to produce foundational artistic structures, images that mimic the style of Claude Monet [7]. Inspired by
though further refinements are necessary for achieving higher the kaggle competition I’m Something of a Painter Myself
levels of realism and fidelity to Monet’s style. Future work focuses [8], the proposed approach utilizes a tiered GAN model to
on improving training methodologies and model complexity to
bridge the gap between generated and true artistic images. Addi- progressively refine image quality. Starting from random noise,
tionally, the limitations of traditional GANs in artistic generation the model generates images through multiple stages, with
are analyzed, and strategies to overcome these shortcomings are each stage producing outputs that are increasingly detailed
proposed. and stylistically accurate, as shown in Figure 2. This multi-
Index Terms—Artificial Intelligence, Deep Learning, Image stage process ensures that the G learns to replicate Monet’s
Processing, Generative Adversarial Networks (GANs), Artistic
Image Generation, Visual Interpretation
distinctive style, enabling the creation of high-quality artistic
images [9], [10].
The key contributions of this work are as follows:
I. I NTRODUCTION • The proposed tiered GAN model employs multiple GANs
in succession to progressively enhance image quality,
Generative Adversarial Networks (GANs) have gained sig- transforming low-quality inputs into refined representa-
nificant attention in machine learning, particularly in appli- tions of Monet’s style.
cations such as image generation, style transfer, and data • An efficient training method is introduced, incorporating
augmentation. Introduced by Ian Goodfellow in 2014, GANs downsampling and convolutional layers to enable high-
consist of two neural networks—a generator (G) and a dis- quality artistic generation while optimizing computational
criminator (D)—engaged in a zero-sum game [1], [2]. The resources.
generator (G) creates synthetic data, while the discriminator • An experimental analysis of GAN limitations in artistic
(D) evaluates its authenticity relative to a given dataset, style generation is provided, highlighting the shortcom-
such as Monet paintings [3]. This adversarial process drives ings of traditional GAN architectures and proposing po-
both networks to iteratively improve, enabling G to produce tential improvements.
increasingly indistinguishable outputs (see Figure 1). As a The paper is organized as follows: Section 2 presents the
result, GANs have significantly advanced the generation of background. Section 3 describes GAN architecture. Section
realistic images, videos, and even text [4], [5]. In medical 4 discusses the methodology. Section 5 presents the results
imaging, GANs enhance diagnostic image resolution and and discussion. Section 6 concludes this work with future
generate synthetic data to supplement limited datasets [6]. directions.
However, the complexity of deep learning models poses
challenges in understanding their decision-making processes, II. BACKGROUND
necessitating interpretability and visualization techniques to The field of artistic image generation has progressed sig-
improve transparency [6]. nificantly with the advent of GANs, which excel at creating
 C. GAN Applications in Artistic Image Generation
Generating artistic images from scratch remains challenging
despite advances in style transfer. Neural style transfer, ex-
plored by Gatys et al., applies the artistic style of one image
to another’s content [16], but relies on pre-existing content
images.
ArtGAN, introduced by Tan et al., generates creative images
by learning complex artistic features from training data [17].
It employs classification-aware loss functions to guide diverse
artistic style generation but often struggles with inconsistent
quality for certain styles and requires large datasets.
Creative Adversarial Networks (CANs), proposed by El-
gammal et al., extend GANs to generate novel artwork by
encouraging deviation from existing styles [18]. While capable
Fig. 2: A tiered GAN model: Transforming random input to of originality, CANs can produce overly abstract outputs,
refined output. The left image displays the initial random requiring careful control.
noise, and the right image shows the refined image. This Generating highly detailed and realistic artistic images faces
highlights the model’s ability to enhance and transform images persistent challenges, including instability, mode collapse, and
effectively. balancing the generator and discriminator. Techniques like
Wasserstein GANs (WGAN) [19] and Spectral Normalization
visually compelling images. These models are extensively used [20] have been applied to stabilize training and enhance image
in tasks such as style transfer and image refinement. This diversity. This work incorporates these techniques to generate
section explores key advancements in GAN architectures, em- Monet-style images from scratch.
phasizing their evolution from domain-to-domain translation The complexity of fine art, characterized by detailed textures
to the creation of original artistic content. and intricate color compositions, adds to the challenges for
GAN-based models. To address this, the proposed tiered GAN
A. GANs for Image Generation and Style Transfer system uses multiple GANs in succession to progressively re-
fine outputs, improving quality incrementally. Inspired by Amy
Conditional GANs, introduced by Mirza et al., enable image Jang’s Monet CycleGAN tutorial [21], which demonstrates
generation conditioned on specific input variables, such as style transfer using unpaired datasets, this work generates
class labels [11]. This extension allows for controlled out- Monet-style images directly from random noise rather than
puts but faces stability challenges, particularly with noisy or transforming existing images.
unbalanced data. Building on the concept of progressive refinement intro-
CycleGAN, developed by Zhu et al., facilitates unpaired duced by ProGAN [14], they proposed multi-stage GAN
image-to-image translation, enabling style transfer without model that transforms random noise into detailed Monet-
requiring paired datasets [12]. While effective, CycleGAN style images through collaborative refinement. Unlike image-
can distort image content during translation, affecting fi- to-image translation models such as CycleGAN; this approach
delity. StyleGAN, proposed by Karras et al., generates high- focuses on generating original content from scratch.
resolution images with adjustable styles by disentangling style Architectural choices were influenced by simpler datasets,
and content [13]. Despite its success, StyleGAN demands such as Fashion MNIST [22], where GANs are effectively used
significant computational resources and extensive training, to generate basic grayscale images. This work extends these
limiting accessibility. principles to create detailed and stylistically accurate Monet-
inspired artwork.
B. Progressive GANs and Image Refinement
III. G ENERATIVE A DVERSARIAL N ETWORKS (GAN S )
Progressive Growing of GANs (ProGAN), introduced by A RCHITECTURE
Karras et al., improves training stability by starting with GANs are a class of generative models consisting of two
low-resolution images and progressively growing to higher neural networks, namely Generator (G) and Discriminator
resolutions [14]. This approach enhances detail in generated (D), trained simultaneously through adversarial learning. The
images but increases training time and resource demands as primary objective of GANs is to generate synthetic data
resolution grows. samples that are indistinguishable from real data samples.
Super Resolution GAN (SRGAN), proposed by Ledig et al.,
targets image super-resolution by generating high-resolution A. Generator (G)
images from low-resolution inputs while maintaining percep- A Generator network, denoted as G, maps a noise vector
tual quality [15]. However, SRGAN can produce artifacts, z ∈ Rd to a data sample xfake ∈ Rn . The input noise vector
particularly in complex textures. z is typically sampled from a simple prior distribution such
as a Gaussian or uniform distribution pz (z). The mapping is E. Training
mathematically expressed as: The training involves alternating updates to D and G.
Specifically, D is updated to maximize its classification accu-
xfake = G(z; θG ), (1) racy, while G is updated to minimize D’s ability to distinguish
real from fake samples. The training steps are as follows:
where θG represents the learnable parameters of G. The 1) Sample a batch of real data {xi }m i=1 from the data
goal of G is to approximate the true data distribution pdata (x) distribution pdata (x).
by generating realistic samples. 2) Sample a batch of noise vectors {zi }mi=1 from the noise
distribution pz (z).
B. Discriminator (D) 3) Compute D loss LD and update D’s parameters θD
A Discriminator network, denoted as D, acts as a binary using backpropagation.
classifier that distinguishes between real data samples xreal and 4) Compute G loss LG and update G’s parameters θG using
generated (fake) data samples xfake . It outputs a scalar value backpropagation.
D(x) ∈ [0, 1], representing the probability that the input data D’s objective is to maximize:
x is real. The function is defined as:

max Ex∼pdata (x) [log D(x)] + Ez∼pz (z) [log(1 − D(G(z))], (6)
D(x; θD ), (2) D

while G’s objective is to minimize:

where θD represents the learnable parameters of D. The
goal of D is to maximize the probability of correctly identi-
min Ez∼pz (z) [log(1 − D(G(z)))], (7)
fying real data and minimize the probability of misclassifying G
generated data. or equivalently:
C. Adversarial Objective min Ez∼pz (z) [log D(G(z))]. (8)
G
The training process of GANs is structured as a two-player
minimax game between G and D. The following minimax This formulation leads to an equilibrium where G’s distribu-
objective function formalizes this: tion matches the real data distribution, making it challenging
for D to distinguish between real and fake samples.

min max V (D, G) = Ex∼pdata (x) [log D(x)] IV. M ETHODOLOGY

G D (3)
+ Ez∼pz (z) [log(1 − D(G(z)))]. A. Environment
The software was developed using Python 3.9 and Ten-
here, the first term corresponds to D maximizing the log sorFlow framework version 2.16.1. The experiment was con-
probability of correctly identifying real data, while the second ducted on Google Colab with 8 cores, utilizing an NVIDIA
term corresponds to D minimizing the probability of being Tesla V100 GPU and 32 GB of RAM. The environment was
deceived by fake samples. supported by CUDA 11.8.0 and cuDNN version 8.8.0.

D. Loss Functions B. Data Preprocessing

The loss functions for both networks are derived from the A dataset of RGB images with dimensions (256, 256, 3) was
minimax objective. resized to (128, 128, 1) using nearest-neighbor interpolation.
1) D Loss: The loss function for D is defined as: This resizing approach retained the recognizable features of
the images, ensuring compatibility with our model’s require-
ments at the chosen resolution. The dataset used, monet jpg,
LD = −Ex∼pdata (x) [log D(x)] consists of 300 Monet paintings sized 256 × 256 in JPEG
(4)
−Ez∼pz (z) [log(1 − D(G(z)))] format and is publicly available on kaggle [23].

This encourages D to output higher probabilities for real C. Model

samples and lower probabilities for generated samples. The initial GAN model utilized random noise as input to
2) G Loss: The loss function for G is defined as: generate full-size Monet paintings, establishing a foundation
for further experimentation. Figure 3 illustrates the Monet
LG = −Ez∼pz (z) [log D(G(z))]. (5) GAN architecture, featuring both G and D, as shown in Figure
4. These are sequential models incorporating convolutional
This objective encourages G to produce data that maximizes and dense layers. Leaky ReLU is employed as the activation
D’s output, effectively fooling D into classifying fake data as function in the hidden layers, while the output layer of D uses
real. sigmoid activation for binary classification.
 progressive refinement from M3 to M2, M2 to M1, and M1
to MF, as shown in Figure 5.
E. Training the Tiered GAN System
The initial experiment aimed to generate Monet-style im-
ages from random noise using an input tensor of size
(128, 128, 1). This experiment was conducted over 500
epochs, with a total training time of approximately 3 hours.
Despite this effort, the results were unsatisfactory, as the
generator (G) consistently produced black images. The limited
number of training epochs, the relatively small dataset of
300 Monet paintings, and insufficient training time likely
contributed to the model’s poor performance, revealing the
Fig. 3: Different steps of Monet GAN Model
need for a more robust training approach.
To address these issues, a primary tiered GAN system was
implemented, involving the sequential training of four separate
GANs. Each GAN was trained for 2000 epochs, with the
duration per epoch ranging from 22 to 27 seconds. Leveraging
parallel processing on an NVIDIA Tesla V100 GPU with 8
CPU cores in the Google Colab environment, the model was
able to efficiently perform computations such as convolutional
operations and backpropagation. This parallelism reduced the
overall training time, enabling the completion of 2000 epochs
per GAN in approximately 12 hours, a significant improve-
ment in computational efficiency. The testing phase, which
involved generating images, took approximately 5 minutes
per GAN, yielding outputs that progressively improved with
each tier. This multi-stage strategy allowed the system to
refine the generated outputs iteratively, enabling better learning
from the dataset and overcoming the limitations of the initial
experiment.
Fig. 4: Generator and Discriminator Architecture Parallel processing also facilitated higher scalability, en-
abling the system to handle more complex architectures
D. Input Strategy and Progressive Refinement and experiments without substantial increases in runtime.
The implementation explored various input strategies for G, The tiered GAN approach, by leveraging optimized training
beginning with random noise as input, which proved highly strategies and sufficient resources, significantly enhanced the
challenging. To address this, a multi-step refinement approach model’s performance and quality of results.
was developed, where G begins with partially constructed To ensure balanced training between G and D, both mod-
images and progressively enhances them. els were designed with relatively shallow architectures. The
learning rate for D was set at one-tenth of G’s learning rate,
This method involved creating four datasets from the orig-
specifically 0.0001 for G and 0.00001 for D. The Adam
inal dataset. These datasets, labeled MF, M1, M2, and M3,
optimizer was used for both models, with binary crossentropy
represent progressively lower-quality grayscale images with
as the loss function and a batch size of 8.
dimensions (128, 128, 1). MF contains the full grayscale image
at half resolution, while M1, M2, and M3 are progressively V. R ESULTS AND DISCUSSION
downsampled versions with decreasing detail at each step.
A. Training Challenges
G refines the image through multiple stages, starting with
random noise or a low-detail input: During training, there were several challenges such as
initial attempts to generate images from random noise yielded
• GAN 1: Generates images for the M3 dataset (lowest unsatisfactory results, indicating the need for extended training
quality). durations and a larger dataset. Memory constraints necessitated
• GAN 2: Refines M3 into M2. smaller input sizes, prompting adjustments to the generator
• GAN 3: Refines M2 into M1. architecture, including the use of convolutional layers for
• GAN 4: Produces the final image, MF. effective downsampling.
The initial version of the model used random noise as the The tiered GAN system, designed to progressively refine
input, targeting the M3 dataset. A subsequent version incor- images, initially failed to produce the desired outputs. Instead
porated non-random, downsampled images as input, enabling of Monet-like paintings, the system generated tensors with
Fig. 5: Progressive Refinement of Image Generation through Multiple Stages. G begins with random noise (M3) and undergoes
iterative enhancements across four GAN models, culminating in the final high-quality output (MF). Each stage represents a
significant improvement in detail and quality, transitioning from M3 to M2, then to M1, before achieving the final image.

(a) Random Noise to M3

Training Loss (b) M3 to M2 Training Loss

Fig. 7: Training Loss for Small Input Vector with Upsampling

(c) M2 to M1 Training Loss (d) M1 to MF Training Loss as suggested by [22]; second, replacing the input dense layers
with convolutional layers to reduce memory usage and support
Fig. 6: Training loss curves for discriminator (d loss) and gen- larger input sizes.
erator (g loss) across four phases(a-d): showing the adversarial Initial attempts with standard Conv2D layers proved inef-
dynamics and progressive model refinement. fective. Drawing on methods from [21] and [24], the approach
was adjusted to use deconvolution layers (Conv2DTranspose)
constant values of 1, highlighting the generator’s inability to for upsampling, along with batch normalization to improve
learn meaningful features. This required a thorough reevalua- training stability and reduce convergence time. Despite these
tion of the model’s architecture and training strategy. Figures modifications, the generated images lacked clarity and artistic
6 show the training loss for each GAN tier, with a high initial resemblance to Monet’s style.
loss, particularly for the generator, which gradually decreases The use of a smaller 128-element input array with upsam-
as both the generator and discriminator learn to balance pling showed better results. While clarity remained a limi-
each other. Specifically, Figure 6c highlights the continued tation, the generated images exhibited recognizable patterns
refinement of the model during this phase, where the generator of light and dark areas, indicating that the model had started
and discriminator show improved synchronization, reducing learning relevant features from the dataset. Figures 7 and 8
loss and producing more coherent outputs. show the training loss and an example of the generated images
for this approach.
B. Model Re-evaluation and Alternative Approaches The generated images were evaluated for their resemblance
Issues were identified in the initial tiered GAN models, to Monet’s style and ability to capture artistic details. Quanti-
particularly with the downsampling convolution layers in tative metrics like training loss monitored model performance,
the generator. To address these challenges, two alternative while human evaluation ensured alignment with Monet’s
approaches were explored: first, using a small array of 128 unique style. This combined approach of automated metrics
random values and upsampling them to the target image size, and human assessment identified areas for further refinement
 Fig. 8: Generated Image from Small Input Vector with Upsampling

in the model’s architecture and training. [8] A. Jang, A. S. Uzsoy, and P. Culliton, “I’m something of a painter my-
self,” https://www.kaggle.com/competitions/gan-getting-started, 2020.
VI. C ONCLUSION AND F UTURE W ORK [9] L. Vela, F. Fuentes-Hurtado, and A. Colomer, “Improving the quality
of image generation in art with top-k training and cyclic generative
This research introduced a tiered GAN architecture to em- methods,” Scientific Reports, vol. 13, no. 1, p. 17764, 2023.
[10] X. Jin, “[retracted] art style transfer of oil painting based on parallel
ploy multiple GANs sequentially for enhancing image quality, convolutional neural network,” Security and Communication Networks,
transforming low-quality images into refined representations of vol. 2022, no. 1, p. 5087129, 2022.
Monet’s style. The training methodology efficiently handled [11] M. Mirza, “Conditional generative adversarial nets,” arXiv preprint
arXiv:1411.1784, 2014.
large images using downsampling and convolutional layers, [12] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image
enabling high-quality artistic generation with limited compu- translation using cycle-consistent adversarial networks,” in Proceedings
tational resources. of the IEEE international conference on computer vision, 2017, pp.
2223–2232.
Experimental results were mixed; while the system showed [13] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “An-
potential, it struggled to fully capture Monet’s intricacies. The alyzing and improving the image quality of stylegan,” in Proceedings of
limited dataset of 300 images likely constrained the model’s the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), June 2020.
ability to learn complex artistic features. A larger and more [14] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing
diverse dataset could improve the model’s performance and of gans for improved quality, stability, and variation,” 2018. [Online].
learning capability. Available: https://arxiv.org/abs/1710.10196
[15] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta,
Future work will address these challenges through three A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic
strategies: (1) using larger datasets, augmented with bootstrap single image super-resolution using a generative adversarial network,”
aggregating (bagging), to enhance prediction stability and in 2017 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), 2017, pp. 105–114.
robustness [25]; (2) employing distributed computing inspired [16] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using
by Firebase for efficient processing and synchronization of convolutional neural networks,” Proceedings of the IEEE conference on
large datasets [26]; and (3) incorporating pre-trained models computer vision and pattern recognition, pp. 2414–2423, 2016.
[17] W. R. Tan, C. S. Chan, H. Aguirre, and K. Tanaka, “Artgan: Artwork
via Transfer Learning to accelerate convergence and better synthesis with conditional categorical gans,” 2017. [Online]. Available:
capture Monet’s artistic style. https://arxiv.org/abs/1702.03410
These approaches aim to refine the tiered GAN system by [18] A. Elgammal, B. Liu, M. Elhoseiny, and M. Mazzone, “Can:
Creative adversarial networks, generating ”art” by learning about
leveraging real-time data handling and distributed computing, styles and deviating from style norms,” 2017. [Online]. Available:
addressing current limitations in computational resources and https://arxiv.org/abs/1706.07068
dataset size. [19] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” arXiv
preprint arXiv:1701.07875, 2017.
[20] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, “Spectral normal-
R EFERENCES ization for generative adversarial networks,” in International Conference
[1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, on Learning Representations, 2018.
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” [21] A. Jang, “Monet cyclegan tutorial,” https://www.kaggle.com/code/
Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020. amyjang/monet-cyclegan-tutorial/notebook, 2020, accessed 23 July
2023.
[2] C. Hu, T. Tu, Y. Gong, J. Jiang, Z. Zheng, and D. Cheng, “Tackling
[22] N. Renotte, “Build a generative adversarial neural network with
multiplayer interaction for federated generative adversarial networks,”
tensorflow and python — deep learning projects,” YouTube, June 2022,
IEEE Transactions on Mobile Computing, 2024.
accessed 18 July 2023. [Online]. Available: https://www.youtube.com/
[3] Y. Chang, “Enhancing super resolution of oil painting patterns through
watch?v=AALBGpLbj6Q
optimization of unet architecture model,” Soft Computing, vol. 28, no. 2,
[23] PodcastPrereamea, “Cyclegan monet paintings,” https://www.kaggle.
pp. 1295–1316, 2024.
com/code/podcastprereamea/cyclegan-monet-paintings/data, 2024.
[4] Z. Cai, Z. Xiong, H. Xu, P. Wang, W. Li, and Y. Pan, “Generative
[24] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing
adversarial networks: A survey toward private and secure applications,”
of gans for improved quality, stability, and variation,” in International
ACM Computing Surveys (CSUR), vol. 54, no. 6, pp. 1–38, 2021.
Conference on Learning Representations, 2018.
[5] P. Sharma, M. Kumar, H. K. Sharma, and S. M. Biju, “Generative ad-
[25] A. Darweesh, A. Abouelfarag, and R. Kadry, “Real time adaptive
versarial networks (gans): Introduction, taxonomy, variants, limitations,
approach for image processing using mobile nodes,” in 2018 6th Inter-
and applications,” Multimedia Tools and Applications, pp. 1–48, 2024.
national Conference on Future Internet of Things and Cloud Workshops
[6] D. Bhati, F. Neha, and M. Amiruzzaman, “A survey on explainable
(FiCloudW). IEEE, 2018, pp. 158–163.
artificial intelligence (XAI) techniques for visualizing deep learning
[26] R. Eltehewy, A. Abouelfarag, and S. N. Saleh, “Efficient classifica-
models in medical imaging,” Journal of Imaging, vol. 10, no. 10, p.
tion of imbalanced natural disasters data using generative adversarial
239, 2024.
networks for data augmentation,” ISPRS International Journal of Geo-
[7] N. Shi, Z. Chen, L. Chen, and R. S. Lee, “Relu-oscillator: Chaotic vgg10
Information, vol. 12, no. 6, p. 245, 2023.
model for real-time neural style transfer on painting authentication,”
Expert Systems with Applications, p. 124510, 2024.