Minor Project Report On VibeSync: AI-Driven Gesture and Emotion- Based Spotify Clone Submitted in partial fulfillment of the requirements for the completion of minor project [ARP 455] Name: Abir Sinha Enrollment Number: 10619011721 Under the supervision of Dr. Atul Tripathi UNIVERSITY SCHOOL OF AUTOMATION AND ROBOTICS GURU GOBIND SINGH INDRAPRASTHA UNIVERSITY EAST DELHI CAMPUS, SURAJMAL VIHAR, DELHI- 110032DECLARATION I hereby declare that the Minor project entitled "VibeSyne: AJI-Driven Gesture and Emotion-Based Spotify Clone" is an authentic record of work completed as requirements of Minor project (ARP 455) in the University School of Automation and Robotics under the supervision of Atul Tripathi. bbw Krabur (Signature of Student) Abir SInha 10619011721 Date; _20-—-2b2y ~ (Signature of Supervisor) x Dp Atul Tripathi Date: 2p == 2094. -ACKNOWLEDGEMENT I would like to express my sincere gratitude to all those who supported and guided me throughout the completion of this project. I am deeply appreciative of the valuable insights and direction provided by my mentors and instructors, whose expertise in machine learning and data analysis greatly contributed to the success of this project. Their continuous guidance and constructive feedback were instrumental in refining the approach and achieving the project's objectives. Additionally, 1 would like to acknowledge the encouragement and support of my colleagues and peers, whose discussions and collaboration enhanced my understanding of the concepts involved. This project has been a rewarding experience, and I am sincerely thankful to all who contributed to its success.About Organization The GGS Indraprastha University-East Campus is committed to providing students with a world-class learning experience, fostering their holistic development. This exemplary campus seamlessly blends aesthetics and technology. As India embraces the fourth Industrial Revolution, the university has taken proactive steps by establishing two new Schools of Studies at the East Campus in Surajmal Vihar, New Delhi. Our institution focuses on cutting-edge fields such as Artificial Intelligence & Data Science, Artificial Intelligence & Machine Learning, Industrial Internet of Things, and Automation & Robotics. The campus boasts impressive facilities, including a centralized auditorium, amphitheater, air-conditioned library, advanced laboratories, sports hall, and more. It’s an exciting hub for learning and innovation! The purpose of the school is to initiate continuous interaction with the industry, sharing the industry experiences, understanding industry needs and providing the required support to the corporate world, as well as opportunities to students to work in alliance with Industry. Facilities for students include well equipped library with books from diversified areas, periodicals and National & International Journals. The academic programmers are designed to enable growth and learning in a highly focused and application-based environment. This is achieved through a combination of formal lectures and hands-on experience in well equipped laboratories and through learning based projects. Creating environment of collaboration, experimentation, imagination and creativity is the goal of the school.information into tabbed sections that are easily switchable without leaving the page, it enhances user experience by decluttering the interface and making information accessible with minimal effort.No. Topic Page No. I. Abstract 4 2 Introduction 56 3. Hardware and Software Requirements 7 4 Problem Statement S11 3. Related Work (Literature Survey) 12-13 6. Major Modules of the Project 14-20 7. Screenshots of IDE/Web Portal/Gesture 21-24 8. References 25‘VibeSync: Al-Driven Gesture and Emotion-Based Spotify Clone is an innovative music player that leverages AI and machine learning to create a hands-free, personalized listening experience. By integrating hand gesture recognition, facial emotion detection, and voice commands, this project transforms traditional music control into an adaptive and intuitive system. Hand gestures are detected using HandTrack,js, enabling tasks like play, pause, track navigation, and volume adjustments, while emotion detection through YOLOvS tailors playlists to match the user’s mood, enhancing personalization. Voice assistance further improves accessibility, allowing users to launch gesture functions seamlessly. The system is built using HTML, CSS, JavaScript, and advanced Al models, providing a responsive Spotify-like interface. This project addresses the limitations of conventional music players, such as the need for physical interaction, by introducing features suitable for multitasking environments. By combining cutting-edge tools and techniques, VibeSyne not only meets the functional requirements of a music player but also sets a benchmark for Al- driven multimedia experiences, showcasing the transformative potential of machine learning in everyday applications. This report outlines the problem, technical implementation, and contributions of VibeSync, paving the way for future advancements. in Al-powered entertainment solutions.‘The rise of AI and machine learning technologies has opened up new possibilities for user interaction, particularly in the entertainment industry. VibeSyne is a project that utilizes gesture recognition, emotion detection, and voice control to enhance the user experience in music streaming. By integrating these technologies into a Spotify clone, users can control the music playback using hand gestures, change tracks based on their mood, and use voice commands to trigger specific functions, Traditional music streaming apps typically rely on manual input methods like buttons or voice commands, but VibeSyne brings an innovative approach by adding intuitive, hands-free controls through gestures. The system is built to recognize a set of predefined hand gestures using the HandTrack,s library, while facial recognition algorithms identify the user's emotion to select songs that match their current mood. Key aspects of the project includ 1. Motivation As the demand for more intuitive and efficient user interfaces grows, the traditional touch-and- click approach to interacting with music streaming services seems increasingly outdated. Users often find themselves needing to interact with their devices in environments where physical touch may be inconvenient, such as during exercise, driving, or while multitasking. This motivated the need for a more seamless and user-friendly interface for controlling music. VibeSyne was born out of this idea — to allow users to interact with the Spotify clone using natural gestures, emotion recognition, and voice commands, ensuring that their music experience is more personalized and hands-free. 2. Importance of Gesture Recognition in Musie Control: Gesture recognition, enabled by technologies like Hand Track,js, plays a pivotal role in making music control more intuitive and efficient. Hand gestures allow users to interact with the system in a way that feels natural and fluid. For instance, raising a hand to change the volume or making a fist to pause or play the music becomes much simpler than manually pressing buttons or touching a screen. With the increasing accessibility of devices with built-in cameras, gesture recognition is becoming a viable and user-friendly option for interaction, 3. Emotion-Based Music Selection: The introduction of emotion-based music selection is another key feature of VibeSyne. Emotions have long been known to influence music preferences — for instance, a person who is feeling happy might prefer upbeat tracks, while someone feeling melancholic might prefer slower, more soothing 5tunes. By using YOLOvS, a powerful object detection model, to analyze facial expressions and detect. emotions, VibeSync adjusts the music playlist accordingly. The model can determine whether the user's mood aligns with specific emotional cues, creating a personalized music experience that adapts to the user's current state of mind, How VibeSyne Works: At its core, VibeSyne functions as a clone of the popular Spotify platform. It includes all the essential features of a music streaming service, such as: + Track Navigation: Skip, pause, or go back to previous tracks. + Volume Control: Increase or decrease the volume. + Playlists: Create, modify, and manage playlists. + Emotion-Based Playlist Generation: Create dynamic playlists based on the user's detected mood. The main difference lies in the integration of advanced AT functionalities. Users can interact with the platform through: 1. Gestures: Using the HandTrackjs library, the webcam detects specific hand gestures to control the player. 2. Emotions: Using the YOLOvS model, the system interprets the user’s facial expressions to adjust the music selection based on their emotional state.HARDWARE AND SOFTWARE REQUIREMETS Hardware Requirements: ‘Webcam/Camera: A camera capable of capturing video at a decent frame rate (minimum 30 fps) is required for gesture and emotion detection. Computer/PC: Any system with at least 4 GB RAM, 22.0 GHz processor, and an internet connection for accessing external libraries and running the project smoothly. ‘Microphone (Optional): For integrating voice control, a microphone can be used to capture user voice commands, Software Requirements: ‘Web Browser: A modern web browser (Chrome, Firefox, or Edge) that supports WebRTC and WebSocket communication for webcam access and real-time interactions Operating System: Windows, macOS, or Linux-based operating systems. Text Editor/IDE: Visual Studio Code, Sublime Text, or any other code e the JavaScript, HTML, and CSS files. Libraries/Technologies: ‘+ JavaScript: For implementing the functionality of the web application, controlling the audio playback, and integrating gesture control ‘+ HandTrack,js: A machine learning library used for real-time hand gesture detection. + YOLOvS: A machine learning model for detecting emotions using facial recognition. ‘+ WebSocket: For real-time communication between devices for features like voice control or networked gesture control ‘+ HTML/CSS: For building the user interface of the Spotify clone. ‘+ TensorFlow.js: For additional machine learning models or integration with the emotion detection functionality. ‘+ Font Awesome: For providing various icons used in the user interface. ‘+ Nodejs: For handling server-side logic, including the management of WebSocket ‘connections to write and executePROBLEM STATEMENT ‘The rise of music streaming platforms has revolutionized the way users access and interact with music. However, despite advancements in technology, the interfaces of these platforms remain relatively static and one-dimensional, This project aims to address three primary challenges in existing music streaming systems: the lack of hands-free interaction, limited personalization, and a ‘monotonous user interaction experience. 1 tations of Traditional Musie Control Mechanisms Current music streaming platforms rely heavily on traditional input methods such as touch, swipe, and voice commands. While these mechanisms are effective in controlled environments, they fi to provide a fully hands-free and seamless experience, particularly in situations where physical interaction is restricted Examples include: ‘+ Driving: Users cannot afford to divert their attention to touch controls or voice commands while focusing on the road, + Exercising: Physical exertion makes using touch-based interfaces inconvenient or impossible. ‘+ Cooking: Hands are often occupied or unclean, making touch interaction impractical. ‘+ Public Places: Noise and privacy concerns limit the effectiveness of voice commands. The absence of a truly intuitive, hands-free music control system creates significant barriers for users in such scenarios, diminishing the overall user experience. 2. Lack of Emotional Context in Personalization While modern platforms use advanced recommendation algorithms based on user preferences and historical data, they rarely consider the user's emotional state or current context. Music has a profound emotional impact, and the ability to adapt playback based on a user's mood can significantly enhance engagement. Current Limitations in Personalization: ‘+ Platforms suggest songs primarily based on listening history, playlists, and genres but fail to adapt in real time. ‘+ Emotional disconnect occurs when a user's current mood does not align with the algorithm- generated playlist. For instance, a user feeling stressed might prefer calming tracks, but the platform may recommend energetic tracks based on past preferences. By integrating emotional intelligence into music control systems, platforms can provide a 8dynamic and deeply personalized experience that aligns with the user’s current emotional context. 3. Repeti ¢ and Non-Immersive Interaction Modern interfaces often rely on repetitive actions such as tapping, swiping, and clicking, which can become mundane over time. These methods fail to enhance the sense of immersion or engagement for the user. Impact on User Engagement: ‘+ Limited novelty in interaction can lead to reduced enthusiasm for using the platform. ‘+ Repetitive gestures do not leverage the full potential of advanced technologies like gesture recognition or Al-based interaction Creating a music control system that integrates intuitive, interactive, and context-aware features can elevate the user experience by making it more immersive and enjoyable. Proposed Solution: Context-Aware, Hands-Free Music Control System This project aims to develop an interactive, context-sensitive music control system that overcomes the limitations of traditional platforms. The proposed system leverages advanced technologies such as gesture recogniti mm, Al-based emotional analysis, and context-awareness to create a dynamic and user-friendly experience. Key Features Include: 1. Gesture-Based Interaction ‘+ Hands-free operation using gestures for play, pause, skip, and volume adjustment, ‘+ Integration with computer vision technologies for accurate gesture detection. ‘+ Customizable gestures to match user preferences. 2. Emotional Intelligence and Context Awareness ‘+ Real-time emotional analysis using facial expression recognition or wearable sensors, ‘+ Contextual adaptation of music recommendations based on mood, activity, or time of day. ‘+ Dynamic playlist generation aligned with the user's emotional state. 3. Seamless Integration with Daily Activities ‘+ Designed for hands-free interaction in scenarios like driving, exercising, and cooking, ‘+ Integration with smart devices (¢.g., smartwatches, AR/VR headsets) for a connected experience. ‘+ Offline functionality for uninterrupted usage in remote or noisy environments.Enhaneed Immersion ‘+ Use of augmented reality (AR) or virtual reality (VR) for a more immersive music experience, ‘+ Dynamic visualizations and interactive environments synchronized with the musi ‘Technical Advantages of the Proposed System Enhanced Accessibility: By removing the dependence on traditional input methods, this system ensures greater accessibility for users with physical limitations or situational constraints, AI-Driven Personalization: ‘+ Incorporates machine learning algorithms for adaptive playlist recommendations. + Combines historical data with real-time inputs for a holistic user profile. |. Scalability and Integration: ‘+ Compatible with a wide range of devices and operating systems. ‘+ Potential for integration with popular music streaming platforms like Spotify, Apple Music, and YouTube Music. Energy Efficiency: Optimized algorithms to minimize computational requirements, ensuring low power consumption on mobile devices. Potential Impact and Applications ‘The proposed solution has far-reaching applications beyond just music streaming. By creating a context-aware and interactive control system, the technology can be extended to: Healthcare: Music therapy sessions can adapt to patients’ emotional states for improved therapeutic outcomes. Gaming: Interactive music controls enhance immersion in gaming environments. Education: Adaptive learning environments that use music to aid focus or relaxation.Challenges and Future Directions ‘While the proposed system offers numerous benefits, several challenges need to be addressed: Gesture Recogni m Accuracy: Ensuring high accuracy across diverse user demographics and environmental conditions Privacy Concerns: Protecting user data, particularly for emotional analysis and context- aware recommendations. Device Compatibility: Balancing advanced features with compatibility for low-end devices. Future enhancements could include: Integration with wearable technology for seamless emotional analysis. Development of voice and gesture hybrids for a more robust hands-free experience. Explora n of adaptive AR/VR interfaces to redefine interactive music experiences.RELATED WORK (LITERATURE SURVEY) In recent years, advancements in gesture recognition, emotion analysis, and voice interaction technologies have significantly enhanced the development of user-centric entertainment systems. ‘This section reviews existing works in gesture-controlled music players, emotion-based music recommendations, and voice-controlled music systems, highlighting their contributions, limitations, and relevance to projects like VibeSyne. Gesture-Controlled Music Players: Gesture recognition has transformed how users interact with digital devices, and its application in music systems is a prime example. Early innovations in this area focused on specialized hardware, such as the Leap Motion Controller, which allowed users to perform gestures in three-dimensional space. This device facilitated intuitive interaction, enabling users to control playback, skip tracks, and adjust volume. However, reliance on proprietary hardware posed a barrier to widespread adoption, as it limited accessibility and increased costs. ‘To overcome these challenges, software-based solutions such as HandTrack,js have gained traction. HandTrack.js leverages computer vision techniques to detect and track hand gestures using a standard webcam. Unlike hardware-dependent systems, this approach democratizes gesture recognition by removing the need for additional devices. HandTrack.js has proven to be effective for applications like VibeSync, as it ensures a seamless, hardware-independent user experience. Despite these ‘advancements, gesture systems are often constrained by environmental factors, such as lighting conditions and background noise, which can affect the accuracy of gesture detection, Emotion-Based Music Recommendat Emotion-aware systems have become an integral part of modem music recommendation platforms. By understanding the emotional state of users, these systems curate personalized playlists that align with their mood, creating a more immersive and satisfying experience. Facial Expression Recognition (FER) is a widely researched technique in this domain, For example, a notable study demonstrated the integration of YOLOvS, a deep learning model traditionally used for object, detection, for real-time emotion recognition. YOLOvS's speed and accuracy make it well-suited for applications that require instantaneous feedback, such as emotion-aware music systems. Platforms like Moodagent and Flow Music have incorporated similar concepts, dynamically creating playlists that adapt to user emotions. These platforms use a combination of machine learning and. emotion analysis to deliver recommendations, improving user engagement and satisfaction. However, such systems face challenges, including the need for high-quality facial data and potential privacy concerns associated with real-time emotion tracking. In addition to facial recognition, some systems have explored multimodal approaches, combining voice tone analysis, physiological signals (like heart rate), and text sentiment to refine emotion detection. These approaches offer a more holistic understanding of user states but require complex integrations and higher computational power. Voice-Controlled Music Systems:Voice interaction has revolutionized how users control music systems, with platforms like Google Assistant, Amazon Alexa, and Apple Siri leading the way. These systems rely on sophisticated Natural Language Processing (NLP) models to interpret and respond to user commands. By analyzing spoken language, these assistants enable intuitive control over music playback, including playing specific tracks, adjusting volume, and creating playlists ‘The integration of voice control into projects like VibeSync introduces new dimensions of user interaction, While gesture-based controls offer tactile engagement, voice commands provide a hands- free alternative, enhancing accessibility for diverse user groups. However, voice-controlled systems ‘are not without limitations. Background noise, accents, and language variations can pose challenges for accurate command interpretation. Additionally, continuous listening devices raise concerns about data security and user privacy. In VibeSync, voice interaction is implemented as a complementary feature alongside gesture control, providing users with multiple interaction modes. This hybrid approach addresses the shortcomings of standalone systems by offering a more versatile user experience. Emerging Trends and Innovations: 1, Multimodal Interaction Systems: Recent research emphasizes the integration of multiple input modalities, such as gestures, voice, and facial expressions, to create adaptive and inclusive user experiences. For example, combining gestures with voice commands can provide redundancy, ensuring system usability even when one input method fails. 2. Edge Computing for Real-Time Applications: The growing need for real-time interaction in gesture and emotion-based systems has driven the adoption of edge computing. By processing data locally, systems like VibeSync can achieve lower latency and enhanced performance, making them ideal for dynamic environments. 3. Augmented Reality (AR) Interfaces: Gesture recognition is increasingly being integrated into AR platforms, where users can interact with virtual music players in a 3D environment. This innovation holds promise for creating immersive entertainment experiences that blend physical and digital elements 4. Deep Learning in Gesture and Emotion Recognition: Advances in deep learning algorithms, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNS), have improved the accuracy of gesture and emotion detection. These models enable systems to learn complex patterns from diverse datasets, enhancing their adaptability and performance.Major Modules of the Project 1. User Interface (UD: ‘The User Interface (Ul) is a crucial aspect of the system, serving as the first point of interaction for the user. It mimies Spotify’s functionality to ensure familiarity and ease of use, providing users with an intuitive experience. Purpose: The Ul is designed to be visually appealing and user-friendly, enabling users to navigate the musie player effortlessly. By offering familiar layouts and features, it ensures a smooth transition for users already acquainted with music streaming applications, HTML, CSS, and JavaScript form the foundation of the front-end. HTML provides the structure, CSS adds styling, and JavaScript enables interactivity + Key Features: ‘+ Aplaylist display showcases song titles, durations, and control buttons for play, pause, next, and previous tracks. ‘+ Avolume control slider is linked to the system’s volume adjustment module, ensuring smooth sound management ‘+ Interactive animations enhance usability, especially for gesture and voice-based actions, giving users a dynamic experience. Why a Spotify Clone? Creating a Spotify clone leverages a familiar design and functionality, making adoption seamless. It also serves as a flexible base for integrating advanced features like emotion detection and gesture recognition. 2. Gesture Recognition Module (HandTrackjs) ‘The Gesture Recognition Module enhances accessibility by allowing users to control the musie player hands-free. Purpose: Th play, pause, skip, and volume adjustment. module detects and interprets hand gestures to map them to music control actions likeImplementation: Technology Used: HandTrack.js, a JavaScript library powered by TensorFlow,js, enables gesture detection. Working Mechanism: ‘+ The webcam captures real-time video feeds, processed by HandTrack,js to identify hand gestures, ‘+ Pre-trained machine learning models detect hand positions and movements, assigning labels to gestures such as "fist" (play/pause), "thumb up" (next track), and “hand tilt" (volume adjustment) ‘Technical Detail ‘The system uses Convolutional Neural Networks (CNNs) for efficient video processing and real- time gesture detection. Why HandTrack.js? ‘+ Lightweight and browser-friendly. ‘¢ No GPU acceleration required, ensuring compatibility with a wide range of devices. ‘+ Simplified API for quick integration into web applications. Benefits: Enables hands-free music control for convenience di ing activities like cooking or driving. ‘Adds a futuristic, interactive dimension to the user experience, 3. Emotion Detection Module (YOLOvs) ‘The Emotion Detection Module personalizes the music experience by adjusting playlists based on the user's emotional state Purpose: Detects emotions such as happiness, sadness, or neutrality through facial expressions, dynamically tailoring playlists to match the mood. Implementation: ‘What is YOLO? YOLO (You Only Look Once) is an advanced object detection framework that processes images in real-time to identify objects and their locations. Why YOLOvS? ‘+ Lightweight and faster than predecessors, making it suitable for real-time applications. ‘+ Optimized for edge devices, balancing performance and hardware efficiency. ‘+ Provides pre-trained models and easy APIs, simplifying the development process,‘Working Mechanism: Input Processing: Frames from the live video feed are resized for YOLOvS’s input requirements Feature Extraction: The model identifies key features, using a backbone network and Feature Pyramid Network (FPN) for multi-scale detection. Output: Detected emotions are mapped to appropriate playlists, providing a customized listening experience. Benefits: Improves user satisfaction by aligning music with emotions, Fast and accurate emotion detection ensures real-time adaptability 4, JavaScript Framework for Core Functionality JavaScript serves as the backbone of the system, integrating all modules seamlessly. Core Features: Listens for events triggered by gestures ot voice commands, Pro es emotion detection outputs to update playlists dynamically. Maintains real-time synchronization between modules. Why JavaScript? Universally supported across platforms. Integrates easily with HandTrack js and TensorFlow.; libraries. Enhances responsiveness and interactivity in web applications, 5. Musie Player Logie This module manages song playback and playlist interactions. Features: Utilizes HTMLS's