Electronics & Communication Engineering

Department
2022-2023

Final Year Project Presentation

Group 59

Title : Multimedia Recommendation System using Machine Learning

Guide : Dr. Sumit Kumar Jha Ayushman Singh

Shubham Sharma
Balvendra Singh
 Abstract
Introduction
Motivation
Literature Survey
Table of Methodology
Content Results
Conclusion
References
Abstract
● Songs, as a medium of expression, have always been a popular choice to depict and understand
human emotion.
● Reliable emotion based classification systems can go a long way in helping us parse their meaning.
● In this project, we present an effective recommendation system which recommends music based on
the real-time mood of the user.
● Music has got unique ability to lift up one's mood. If user receives recommendation based on his
preference, it will also improve his listing experience
● Our music player contains three modules: Emotion Module, Music Classification Module and
Recommendation Module.
● The Emotion Module takes an image of the user's face as an input and makes use of deep learning
algorithms to identify their mood.
● Then with the help of suitable Music Libraries we use the Mood of the user to recommend the songs.
● The Recommendation Module suggests songs to the user by mapping their emotions to the mood
type of the song

1.
 Introduction
● In this project, we suggest a different approach to song recommendation
where their facial expressions detect a person's mood. This approach is
more efficient than the existing ones and eases users' work of first
searching and then creating a specific playlist. For detecting a person's
mood, the facial expression is a crucial point. A webcam or camera is used
to picture a face, and input is extracted from that picture. This input is also
used for determining an individual's mood.

● Based on a person's mood system, prepare a music catalog that eliminates

manual efforts to create a playing list and segregate or group songs into
different lists. This system helps in generating an appropriate playlist
according to a person's mood. An mood -based music recommendation
system aims to scan and interpret the user's emotions and create a playlist
based on the parameters. Thus, our proposed system concentrates on
identifying human emotions for creating an emotion-based music player.
 Introduction
⮚ Depression is one of the prominent states of illness that has increased its
impact everywhere. But, unfortunately, this state of mind cannot only be
treated with medication but also requires a mood enhancer. Moreover, a
depressed person cannot be left alone as it induces suicidal threats. Thus, I
wish that I could help such people using technology.
⮚ If the technology is used in a better and productive way , suicides and
many other problems originating because of depression can be treated
more easily and quickly.
⮚ Human needs different types of music in every other situation. At times of
physical work, he wants different music. When a person wants to relax, he
wants to listen to different types of music.
⮚ So, the music recommendation system can be applied in different areas like
physical work, stress, music therapy, studying, etc. In this work, we
introduce the model of the customized emotion-driven music
recommendation system. This project clarifies the approach of applying
.
emotion-driven personalization during 3the music recommendation process
 Motivation
The main motivation of this paper is to design
a effective music player which automatically
generates a sentiment aware playlist based on
50% Service Description

the emotional state of the user. The emotion

module determines the emotion of the user.
Relevant and critical audio information from a
song is extracted by the music classification
module. The recommendation module
32% Service Description

combines the results of the emotion module

and the music classification module to
recommend songs to the user. This system
provides significantly better accuracy and
performance than existing systems.
70% Service Description
Literature Survey
The work done in [1] is very helpful in giving us idea about The music classification module in particular,
performs significantly well; it achieves high accuracy in the “angry” category, while also performing appreciably
well in the “happy” and “calm” categories. Thus by efficiently mapping the user's emotion to the correct song class
with an overall accuracy of 97.69%, it achieves optimistic results for the four moods studied. [2] proposed a
system for music recommendation based on human emotion(facial expression). Proposed system consists of two
consecutive CNN models. First CNN model was used for detecting seven different categories of emotions, and
second CNN model recommending music based upon those emotions. [3] describe a novel music recommendation
technique based on the identification of personality traits, moods, and emotions of a single user, starting from solid
psychological observations recognized by the analysis of user behavior within a social environment. [4] utilizes
two well-known algorithms, namely Yoloface and Deepface, wherein the former extracts the face from the live
image and the latter classifies the type of emotion expressed by the person. The output of the Deepface algorithm is
provided as an input to the music recommendation application. This application plays songs based on the category
of emotion predicted by the Deepface algorithm.
 Workflow
Flow Diagram of the Proposed System
No

Face
Yes Extract Related Yes
Start Read Video Detected Information Song Result
?
Of Face Found

No
Search in
Song
Database
 Workflow
Face Webcam Face Detection Process
Flow Chart No

Initialising the Process

Face
Yes Extract Related Yes
Start Read Video Detected Information Song Result
?
Of Face Found

Face Detected
No No
Search in
Song
Database
Emotion Detected

Stop
Workflow
Emotion Detection
End

Face Analyse
Face Face Pre-Processing image
Found Of Image
Detection Cropping Using CNN
?

Emotion
Methodology
⮚ The complete system divides into three parts

i. face detector,

ii. emotion detection from face

iii. music recommendation system based on the mood.

⮚ We use HAAR cascades for face detection. First, we take the picture from the
webcam. Then HAAR cascades scan the complete image and return the
position of detected faces as Rect (x,y,w,h).
Methodology

⮚ We design CNN model for emotion detection and K- Cluster Model for Music Recommendation.

⮚ This model classifies the image into Three classes: neutral, sad, happy.

⮚ We have used Unsupervised Machine Learning techniques to cluster songs into two distinct classes:
'VERY ENTERTAINING' (class 0) and 'RELAXED' (class 1). This clustering is achieved using the widely-
used K-means algorithm. The recommended songs are then ordered based on their current
popularity.

⮚ We have an unique story in the way we recommend the songs for each mood, for example when
other sites recommend sad songs when a person is sad or feeling bad, we recommend users with
songs which will cheer them up('VERY ENTERTAINING') and 'RELAXING' songs when they are 'HAPPY'.
 Methodology

● Details regarding Data Set.

● The data we have used here is from
https://www.kaggle.com/yamaerenay/spotify-dataset-19212020-160k-t
racks
.
● The file contains more than 160,000 songs collected from Spotify Web
API, here the data can be classified by artist, year or genre in the
data.
● It has features like energy, loudness and danceability etc which make
the clustering algorithm work more effectively.
 Methodology

● Data set used was

● Division of Training and Testing data- 80 % data for training
● 20 % data for testing.
● Accuracy of training model 0.9992979288902262 and the
● Accuracy of testing model was 0.9946141116293065.
CNN(Convolutional Neural
Network)
A CNN is specifically designed for analyzing visual data, such as images, and utilizes various layers,
including convolutional layers, pooling layers, and fully connected layers. Convolutional layers extract
features from the input image by applying filters, while pooling layers reduce the dimensionality of the
extracted features. The fully connected layers are responsible for the final classification based on the
extracted features
K-Means Clustering
The K-means algorithm is a popular unsupervised machine learning technique used for
clustering data. It aims to partition a given dataset into K distinct clusters, where each data point
belongs to the cluster with the nearest mean (centroid). The algorithm works iteratively by
initially assigning random centroids to the clusters and then repeatedly updating the centroids
based on the mean of the data points within each cluster. This process continues until
convergence is reached, where the centroids stabilize and the clustering is considered optimal.
Methodology
⮚ Our music recommendation system takes input from the user, where they
can select the preferred singer whose music they want to listen to.
⮚ The system is equipped with a face detection algorithm that detects faces of
the user in front of the camera and marks it in a square box.
⮚ Our system is also integrated with emotion detection algorithms, which
detect various moods of the person whose face is visible in the camera.
⮚ After detecting the current mood of the user, the system processes some of
the top songs of the selected singer from Spotify.
⮚ Based on the detected mood, the system recommends a list of songs that are
suitable for the user's current mood and preferences.
Results
Front End Part of Our Model
Results For Emotion
Detection
Results of Music Recommendation
 Conclusion
➢ In summary, the project successfully implemented a music recommendation
system based on user emotion detection. By leveraging computer vision
techniques and deep learning models, specifically OpenCV, CNN, and DNN, the
system was able to process user images and determine their current mood as
either happy or sad.
➢ By integrating these components, a functional music recommendation system
was developed, capable of providing personalized song recommendations based
on the user's emotional state
➢ By considering the user's emotional state, the system has the potential to greatly
enhance the music listening experience by delivering tailored recommendations
that align with the user's mood.
 Future Work

• The model can be made to accommodate more emotions

• Currently it only classifies Sad, Happy, Neutral emotions.
• Comparing with more models so that it a comparative study
can be made about which set of algorithms are good for this.
References
The following papers were referred to make this project

1. V. P. Sharma, A. S. Gaded, D. Chaudhary, S. Kumar and S. Sharma, "Emotion-Based Music

Recommendation System," 2021 9th International Conference on Reliability, Infocom Technologies
and Optimization (Trends and Future Directions) (ICRITO), Noida, India, 2021, pp. 1-5, doi:
10.1109/ICRITO51393.2021.9596276.
2. S. Gilda, H. Zafar, C. Soni and K. Waghurdekar, "Smart music player integrating facial emotion
recognition and music mood recommendation," 2017 International Conference on Wireless
Communications, Signal Processing and Networking (WiSPNET), Chennai, India, 2017, pp. 154-158,
doi: 10.1109/WiSPNET.2017.8299738.
3. G. M, P. A, S. V, S. M and L. K, "Improved Facial Emotion Recognition using Yolo and DeepFace for
Music suggestion," 2022 3rd International Conference on Electronics and Sustainable Communication
Systems (ICESC), Coimbatore, India, 2022, pp. 1124-1127, doi: 10.1109/ICESC54411.2022.9885456.
 Reference
4. O. Ghosh, R. Sonkusare, S. Kulkarni and S. Laddha, "Music Recommendation System
based on Emotion Detection using Image Processing and Deep Networks," 2022
2nd International Conference on Intelligent Technologies (CONIT), Hubli, India, 2022,
pp. 1-5, doi: 10.1109/CONIT55038.2022.9847888.
5. V. Moscato, A. Picariello and G. Sperlí, "An Emotional Recommender System for
Music," in IEEE Intelligent Systems, vol. 36, no. 5, pp. 57-68, 1 Sept.-Oct. 2021, doi:
10.1109/MIS.2020.3026000.
6. A. Dureha, "An Accurate Algorithm for Generating a Music Playlist based on
Facial Expressions", IJCA, vol. 100, no. 9, pp. 33-39, Aug. 2014.
7. R. Kumar and Rakesh, "Music Recommendation System Using Machine
Learning," 2022 4th International Conference on Advances in Computing,
Communication Control and Networking (ICAC3N), Greater Noida, India, 2022, pp. 572-
576, doi: 10.1109/ICAC3N56670.2022.10074362.
Thank you !