A Project Report On

“ Digital Bona-Fide Generator ”

In Partial Fulfilment of the requirement for the award of the diploma in

Computer Engineering

Submitted by:
Vedika Gawande Exam Seat No:
Samiksha Bodkhe Exam Seat No:
Priti Boralkar Exam Seat No:
Rashi Nirale Exam Seat No:

Under the guidance of Prof

S. S. Mete

1
 DEPARTMENT OF COMPUTER ENGINEERING
Government Polytechnic, Yavatmal
2023-24
Examiner Certificate
Project Entitled
By

“Computer Automation Using Hand Gesture Recognition”

Submitted by
Ashwin V. Bondre Exam Seat No: 314852
Atharva S. Mandgaonkar Exam Seat No: 314847
Rushikesh R. Sonule Exam Seat No: 314887
Trishul H. Gawande Exam Seat No: 314859
Yatharth S. Rahangdale Exam Seat No: 314892

Is presented and approved for the Diploma in

Computer Engineering
Of
MAHARASHTRA STATE BOARD OF TECHNICAL EDUCATION,
YAVATMAL

Internal Examiner External Examiner

Name: Name:

2
Date: Date:

CERTIFICATE

This is to certify that the project report entitled

“Computer Automation Using Hand Gesture

Recognition”

Has been duty fully completed by the following students under my guidance, in a
satisfactory manner as partial fulfilment in the diploma course in
Computer Engineering

MSBTE MUMBAI

Submitted by:

Ashwin V. Bondre. Exam Seat No: 314852

Atharva S. Mandgaonkar. Exam Seat No: 314847
Rushikesh R. Sonule. Exam Seat No: 314887
Trishul H. Gawande. Exam Seat No: 314859
Yatharth S. Rahangdale. Exam Seat No: 314892

Name & Signature of Guide- Telephone -

3
________________________ ________________

CERTIFICATE

This is to certify that the project report entitled

“Computer Automation Using Hand Gesture

Recognition”

Has been duty fully completed by the following students under my guidance, in
a satisfactory manner as partial fulfilment in the diploma course in
Computer Engineering

MSBTE MUMBAI

Submitted by:

Ashwin V. Bondre. Exam Seat No: 314852

Atharva S. Mandgaonkar. Exam Seat No: 314847
Rushikesh R. Sonule. Exam Seat No: 314887
Trishul H. Gawande. Exam Seat No: 314859

4
 Yatharth S. Rahangdale. Exam Seat No: 314892

Signature of HOD - Signature of Principle -

________________ ________________

ACKNOWLEDGMENT

We are pleased to present “Computer Automation Using Hand Gesture Recognition”

project and take this opportunity to express our profound gratitude to all those people who
helped us in completion of this project.

We thank our college for providing us with excellent facilities that helped us to complete and
present this project. We express our immense gratitude to our honorable principal sir Dr. R.P.
Mogre. Also, we thanks to our head of the department Prof. S. S. Mete.

We would also like to thank the staff members and lab assistants for permitting us to use
computers in the lab as and when required.

We express our deepest gratitude towards our project guide Prof. S. S. Mete for her valuable
and timely advice during the various phases in our project. We would also like to thank her
for providing us with all proper facilities and support as the project co- coordinator. We
would like to thank her for support, patience and faith in our capabilities and for giving us
flexibility in terms of working and reporting schedules.

We would like to thank all our friends for their utmost important moral support. Finally, we
would like to thank everyone who has helped us directly or indirectly in our project.

5
Ashwin V. Bondre
Atharva S. Mandgaonkar
Rushikesh R. Sonule
Trishul H. Gawande
Yatharth S. Rahangdale

CONTENT PAGE
CHAPTER NO. TITLE PAGE NO.

ASTRACT 7
1 INTRODUCTION 8

1.1 Introduction 9

1.2 Aims 12

1.3 Objective 12
2 LITERATURE SURVEY 13

2.1 Challenges and Future Directions 15

3 SCOPE OF THE PROJECT 16

4 METHODOLOGY 19

5 DETAILS OF DESIGNS, WORKING AND 22

PROCESSES

5.1 Design 23
5.2 Working 26
5.3 Process 28

6 RESULTS AND APPLICATIONS 29

6.1 Result 30

6.2 Application 36

6
 7 CONCLUSION AND FUTURE SCOPE 37
7.1 Conclusion 38

7.2 Future scope 38

REFERENCES 39

ABSTRACT :

This study of project entitled “Computer Automation using Hand Gesture Recognition”
encounters wide range of issues bounded with our day-to-day life. It is very useful tool for
many professionals. It includes recognition of hand gestures by our laptops or pc’s webcam.
The gesture recognized by web cam will be interpreted by the machine learning algorithms
and based on that appropriate action will be taken.

This project includes models like finger counter, volume control, mouse controller, and
keyboard handling. The models will be implemented by using python libraries for machine
learning. The project integrates a gesture recognition component that classifies hand gestures
into predefined categories, enabling users to interact with devices and systems through
intuitive hand movements.

The project aims to provide a user-friendly interface that allows individuals to easily interact
with computers or other devices using hand gestures, reducing the reliance on traditional
input methods. The system is designed to be adaptable and can be integrated into a wide
range of applications, from mobile devices to smart TVs, robotics, and healthcare systems.

By combining computer vision and machine learning technologies, this Hand Recognition
System offers a versatile solution for a variety of practical applications. Its accuracy, real-time
capabilities, and user-friendly interface make it a valuable tool for enhancing human-

7
computer interaction and user authentication while opening doors to innovative gesture

-based control systems in different domains.

8
INTRODUCTION:

9
Hand gesture recognition for human-computer interaction is indeed an area of active research
in computer vision and machine learning (Maung, 2009). One of the primary goals of gesture
recognition research is to create systems that can identify specific gestures and use them to
convey information or control a device. This technology has a wide range of potential
applications, from controlling smart devices and video games to enabling more intuitive
communication between humans and computers. Though, gestures need to be modelled in the
spatial and temporal domains, where a hand posture is the static structure of the hand and a
gesture is the dynamic movement of the hand. Being hand-pose one of the most important
communication tools in human’s daily life, and with the continuous advances of image and
video processing techniques, research on human-machine interaction through gesture
recognition led to the use of such technology in a very broad range of possible applications
(Mitra and Acharya, 2007), of which some are here highlighted:

• Virtual reality: enable realistic manipulation of virtual objects using one’s hands
(Yoon et al., 2006, Buchmann et al., 2004), for 3D display interactions or 2D displays that
simulate 3D interactions.

• Robotics and Tele-presence: gestures used to interact with robots and to control robots
are similar to fully-immersed virtual reality interactions, however the worlds are often real,
presenting the operator with video feed from cameras located on the robot. Here, for example,
gestures can control a robot’s hand and arm movements to reach for and manipulate actual
objects, as well as its movement through the world

• Desktop and Tablet PC Applications: In desktop computing applications, gestures

can provide an alternative interaction to mouse and keyboard. Many gestures for desktop
computing tasks involve manipulating graphics, or annotating and editing documents using
pen-based gestures.

• Games: track a player’s hand or body position to control movement and orientation of
interactive game objects such as cars, or use gestures to control the movement of avatars in a
virtual world. Play Station 2 for example has introduced the Eye Toy (Kim, 2008), a camera

10
that tracks hand movements for interactive games, and Microsoft introduced the Kinect
(Chowdhury, 2012) that is able to track users’ full body to control games.

• Sign Language: this is an important case of communicative gestures. Since sign

languages are highly structural, they are very suitable as test-beds for vision-based algorithms
(Zafrulla et al., 2011, Ong and Ranganath, 2005, Holt et al., 2010, Tara et al., 2012).

Why need of Gesture Recognition Technology...?

Markets and Markets that the gesture recognition market will reach $32.3 billion in 2025, up
from $9.8 billion in 2020. Today’s top producers of gesture interface products are,
unsurprisingly, Intel, Apple, Microsoft, and Google. The key industries driving mass adoption
of touchless tech are automotive, healthcare, and consumer electronics.

Fig. 1.1 Graph of gesture recognition usage

Why may people want to use gestures instead of just touching or tapping a device? A desire
for contactless sensing and hygiene concerns are the top drivers of demand for touchless
technology. Gesture recognition can also provide better ergonomics for consumer devices.
Another market driver is the rise of biometric systems in many areas of people’s lives, from
cars to homes to shops.

11
During the coronavirus pandemic, it’s not surprising that people are reluctant to use
touchscreens in public places. Moreover, for drivers, tapping a screen can be dangerous, as it
distracts them from the road. In other cases, tapping small icons or accidentally clicking on
the wrong field increases frustration and makes people look for a better customer experience.
Realtime hand gesture recognition for computer interactions is just the next step in
technological evolution, and it’s ideally suited for today’s consumer landscape. Besides using
gestures when you cannot conveniently touch equipment, hand tracking can be applied in
augmented and virtual reality environments, sign language recognition, gaming, and other use
cases.

Aim
The aim of the project is to develop a robust system for hand gesture recognition coupled
with a static voice assistant. This system will enable users to control computers and devices
using intuitive hand gestures and voice commands, facilitating hands-free interaction and
enhancing accessibility. By integrating gesture recognition with a voice assistant, the project
aims to create a seamless and efficient interface for various applications, such as virtual

12
mouse control, volume adjustment, and air canvas creation. Ultimately, the project seeks to
explore innovative ways to improve human-computer interaction and foster a more natural

and intuitive computing experience. Objectives

• First objective of this project is to create a complete system to detect, recognize and
interpret the hand gestures through computer vision.
• Second objective of the project is therefore to provide a new low-cost, high speed and
color image acquisition system.
• By automating repetitive or routine tasks, hand gesture recognition systems aim to
increase productivity. Users can perform actions more quickly and efficiently through
intuitive hand gestures, freeing up time for other tasks or activities.

13
Literature Survey:

14
Massive improvement in computer development can be clearly seen involving both
interaction between human and computer itself. Human interaction with computer has
becoming a core component in our daily life as we interact with it throughout the day.
Technology nowadays incorporate flat panel displays as an output device of images being
transmitted electronically. For an instance, most cars are equipped with computerized systems
for navigation and entertainment which needs to be operated and displayed by using a screen
panel. However, controlling a computer by gesturing in the air will be everyone wishes as this
will ease down our tasks. Hand gesture has been visualized as the new evolvement of human
computer interaction which will replace the functions and usage of touch screen displays.

Human computer interaction (HCI) is defined as the relation between the human and
computer(machine), represented with the emerging of computer itself. Vital endeavor of hand
gesture recognition system is to create a natural interaction between human and computer
when controlling and conveying meaningful information. There are two main characteristics
should be considered when designing a HCI system as mention in: functionality and usability.

System functionality referred to a set of actions or services that are provided to its users,
while usability of a system is defined by the level and scope of the system in which the
system can be used efficiently in order to obtain certain specific user purposes. By having an
equal balance between both functionality and usability of a system, an effectiveness and
powerful performance system can be achieved.

However, in order to create a useful hand gesture recognition system, differences between
hand postures and hand gestures need to be distinguished first. Hand posture is a static hand
configuration which is represented by a single image without any involvement of movements,
while hand gesture is defined as a dynamic movement consisting of sequences of hand
postures over a short span of time. For example, making thumbs up and holding it in a certain
position is a hand posture while, waving good-bye is an example of hand gesture.

Challenges and Future Directions

15
• Despite the significant progress that has been made in recent years, there are still
some challenges that need to be addressed before HGR systems can be widely
deployed in real-world applications.
• One challenge is that HGR systems can be sensitive to noise and occlusions. This
means that they may not work well in environments where there is a lot of
background noise or where the user's hands are occluded by other objects.
• Another challenge is that HGR systems can be computationally expensive to train and
run.
• This means that they may not be suitable for use in low-power devices or embedded
systems.
• Despite these challenges, HGR is a promising technology with a wide range of
potential applications. As HGR systems become more accurate, robust, and
affordable, they are likely to be adopted in a wide range of products and services.

16
Scope of the project:

17
Computer automation using hand gesture recognition systems represents a paradigm shift in
human-computer interaction, offering a more intuitive and natural interface for controlling
and interacting with digital devices. With the rapid advancement of computer vision and
machine learning technologies, hand gesture recognition systems have evolved to accurately
interpret and respond to a wide range of hand movements and gestures. This technology holds
immense potential across various domains, including gaming, virtual reality, healthcare,
robotics, and smart environments. By harnessing the power of gestures, these systems enable
users to seamlessly automate tasks, control devices, and access information without the
constraints of traditional input devices. Moreover, hand gesture recognition systems can
enhance accessibility for individuals with disabilities, revolutionize how we interact with
computers in public spaces, and drive innovation in education, entertainment, and beyond. As
research and development in this field continue to progress, the scope for novel applications
and advancements in human-computer interaction is boundless.

• Human Computer Interaction (HCI): Hand gesture recognition systems can

revolutionize HCI by providing a more intuitive and natural interface for interacting
with computers and devices. This includes applications in gaming, virtual reality
(VR), augmented reality (AR), and user interfaces for various software and hardware
systems.

• Healthcare and Rehabilitation: Hand gesture recognition systems have potential

applications in healthcare and rehabilitation settings. They can be used for patient
monitoring, physical therapy, prosthetic control, and assistive technologies for
individuals with mobility impairments.

• Security and Surveillance: Gesture recognition can enhance security and

surveillance systems by enabling more sophisticated monitoring and control
mechanisms. For instance, systems can analyze suspicious gestures or movements for
threat detection, access control, and identification purposes.

• Education and Training: In educational settings, hand gesture recognition systems

can facilitate interactive learning experiences, simulations, and training programs.

18
 They can be used in classrooms, laboratories, and online learning platforms to engage
students and provide hands-on learning opportunities.

• Entertainment and Gaming: The entertainment industry can leverage hand gesture
recognition for immersive gaming experiences, interactive installations, and virtual
environments. Gesture-based controls offer new possibilities for game design, motion
capture, and player engagement.

19
Methodology:
OpenCV is a versatile open-source library used for hand gesture recognition systems. It offers
essential tools for image processing, including feature extraction like contour and edge
detection. Integration with machine learning allows developers to train models for gesture
classification. OpenCV's real-time capabilities are ideal for applications demanding
immediate feedback, such as gaming and human-computer interaction. Gesture tracking
algorithms enable the continuous monitoring of hand movement. Hand gesture recognition
has diverse applications, from touchless device control to accessibility enhancements.

20
OpenCV's active open-source community provides valuable resources, making it a powerful
choice for creating innovative gesture recognition systems.

PROJECT CONSTRAINTS

I propose a vision-based approach to accomplish the task of hand gesture detection. As

discussed above, the task of hand gesture recognition with any machine learning technique
suffers from the variability problem. To reduce the variability in hand recognition task we
assume the following assumptions:

• Single coloured camera mounted above a neutral-coloured desk.

• User will interact by gesturing in the view of the camera.
• Training is must.
• Hand will not be rotated while image is capturing.

The real time gesture classification system depends on the hardware and software

Hardware

• Minimum 2.8 GHz processor Computer System or latest

• .52X CD-ROM drive
• Web cam (For real-time hand Detection)

The hand gesture recognition system has been tested with hand images under various
conditions. The performance of the overall system with different algorithms is detailed in
this chapter. Examples of accurate detection and cases that highlight limitations to the
system are both presented, allowing an insight into the strengths and weaknesses of the
designed system. Such insight into the limitations of the system is an indication of the
direction and focus for future work.

System testing is actually a series of different tests whose primary purpose is to fully
exercise the computer-based system. It helps us in uncovering errors that were made
inadvertently as the system was designed and constructed. We began testing in the 'small'
and progressed to the 'large'. This means that early testing focused on algorithms with
very small gesture set and we ultimately moved to a larger one with improved

classification accuracy and larger gesture set.

21
22
 DETAILS OF DESIGNS, WORKING AND PROCESS:

5.1 Design:

Fig. 5.1.1 Level 0 DFD

The Level 0 DFD is the most basic level of DFD, and it provides a high-level overview of the
entire system. In the image, the Level 0 DFD shows a single process, Request for Service,
and its connection to a single external entity, User. The Request for Service process represents
the entire computer automation system, and the User entity represents the user who is
requesting a service from the system.

Fig. 5.1.2 Level 1 DFD

The Level 1 DFD provides a more detailed view of the system by breaking down the single
process in the Level 0 DFD into sub-processes. In the image, the Level 1 DFD shows the
following sub-processes:
Request for Service: This process receives the user's request for service.

23
Webcam Check: This process checks to see if the user's webcam is turned on and working
properly.
Hands Visibility Check: This process checks to see if the user's hands are visible to the
webcam.
ML Algorithm: This process uses a machine learning algorithm to interpret the user's hand
gestures and determine what service they are requesting.
Process Request: This process processes the user's request and performs the desired service.
Get Service Response: This process retrieves the results of the service request and sends
them back to the user.

24
 Fig. 5.1.3 System Architecture

Fig. 5.1.4 Activity case Diagram for sign Language recognition using hand gesture recognition

Fig. 5.1.5 Use

case diagram

25
5.2 Working
Land Mark Detection.
<<

We’ll first use MediaPipe to recognize the hand and the hand key points. MediaPipe returns a
total of 21 key points for each detected hand.

Fig. 5.2 Finger co-ordinates defined by Mediapipe

These key points will be fed into a pre-trained gesture recognizer network to recognize the
hand pose.

The Fig.5.1.4 is an activity case diagram for a sign language recognition system that uses
hand gestures. It outlines the steps involved in the system, from scanning the hand gesture to
showing the generated output. Here's a breakdown of the steps:

I. Scan Gesture: The hand gesture scanner captures a visual image of the user's hand
sign.
II. Data Pre-Processing: The captured image is then pre-processed to improve the
quality of the data for better gesture recognition. This may involve techniques like
noise reduction, background subtraction, or image normalization.
III. Store Gesture: The pre-processed data is stored in the system's memory.
IV. Traverse Each Data: The system then starts iterating through each piece of data
extracted from the pre-processed image.

26
V. Match: The system checks the extracted data against a database of predefined hand
gestures.
VI. Not Match: If there's no match between the extracted data and the predefined
gestures, the system continues to the next piece of data extracted from the image.
VII. Stop When Delimiter Encountered: The system stops iterating through the data
points when it encounters a delimiter, which is a signal indicating the end of the hand
sign.
VIII. Match: If there is a match between the extracted data and a predefined gesture in the
database, the system moves on to the next step.
IX. Generate Pre-defined Character: The system retrieves the pre-defined character or
word that is associated with the matched hand gesture.
X. Generated Output: The system generates an output, which could be text (word
or character), speech, or any other form of communication corresponding to the
recognized sign language gesture.
XI. Show Generated Output: The system displays the generated output on a screen or
other display device for the user to see.

5.3 Process
• Data Collection: Collect a dataset of hand gesture images or videos with labelled
gestures.
• Pre-processing: Clean, resize, and normalize the data.
• Feature Extraction: Extract relevant features from the images.
• Model Selection: Choose a machine learning or deep learning model.

27
• Model Training: Train the model on the dataset.
• Validation and Testing: Evaluate the model's performance.
• Gesture Labelling: Assign labels to gestures.
• Real-time Data Capture: Capture hand gesture data with a camera or sensor.
• Real-time Inference: Use the trained model for live recognition.
• Feedback and Interaction: Provide feedback or control applications based on
recognized gestures.
• User Interface: Create a user-friendly interface.
• Optimization: Optimize for real-time performance.
• Testing and Evaluation: Continuously test and improve.
• Deployment: Deploy the system in the target environment.

28
 RESULTS AND APPLICATIONS

Result
Voice Assistant:

29
 Fig. 6.1 Launching Voice assistant

This is the output of our static voice assistant SIRI. When we launched the SIRI it greeted us
with respect to the timing condition.

Launching Gesture Recognition via SIRI

30
 Fig. 6.2 Giving command to Voice assistant SIRI
In above output we have launched our one of the module, Virtual Mouse with the help of
SIRI. Searching for Location using SIRI

Fig. 6.3 Launching Google map and searching for location with the help of SIRI

The above output is of launching the location we want with the help of our voice assistant
SIRI.
In above output we have launched the Pune city in Google map.
Virtual Mouse

31
 Fig. 6.4 Launching Virtual Mouse

The above output is of virtual mouse. When we launched the virtual mouse it start’s the
camera to detect the hand

Right click Operation

Fig 6.5 Right click operation of virtual mouse

The above output is of performing the right click operation by using the virtual mouse i.e with
the help of hand gesture without using any mouse
Left click operation

32
 Fig 6.6 Left click operation of virtual mouse

The above output is of performing the left click operation by using the virtual mouse i.e with
the help of hand gesture without using any mouse

Double click operation

Fig 6.7 Double click operation of virtual mouse

The above output is of performing the Double click operation by using the virtual mouse i.e
with the help of hand gesture without using any mouse
Air Canvas

33
 Fig. 6.8 Launching Air canvas

The above output is of air canvas when we launched it.

Drawing operation using Air Canvas

Fig. 6.9 Drawing with the help of air canvas

The above output is of drawing the text or visuals with the help of air canvas.
Volume Control

34
 Fig. 6.10 Increasing volume using Volume controller

The above output is of increasing the volume by using our hand gestures

Fig. 6.11 Decreasing volume using Volume controller

The above output is of decreasing the volume by using our hand gestures.

Applications:
Healthcare:

35
Surgeons can use hand gestures to control medical imaging systems or computer interfaces
during surgeries without needing to touch potentially contaminated surfaces. This reduces the
risk of infections and improves overall hygiene in operating rooms.

Presentations and Lectures:

Presenters and lecturers can use hand gestures to control presentation slides, start/stop videos,
or highlight content without being tied to a remote control or keyboard. This enhances their
ability to engage with the audience while maintaining a smooth flow of the presentation.

Education:

Hand gesture recognition can be integrated into educational tools and platforms to facilitate
interactive learning experiences. Students can use gestures to interact with educational
content, solve problems, and participate in virtual experiments or simulations.

Navigation and Travel:

Voice assistants can provide directions, find nearby restaurants, gas stations, or points of
interest, and help users navigate to their destinations using services like Google Maps or
Apple Maps. Users can ask for directions to a specific location or inquire about traffic
conditions.

Accessibility:

Voice assistants can assist individuals with disabilities by providing hands-free access to
technology and information

36
CONCLUSION AND FUTURE SCOPE

37
Conclusion

The use of virtual whiteboards and hand gesture control for PowerPoint presentations has the
potential to revolutionize the way we teach, learn, and present information. These
technologies offer a more interactive and engaging platform for collaborative learning and
presenting ideas. The virtual whiteboard with gesture control eliminates the need for a stylus
or a mouse, making it more accessible and user-friendly. Similarly, hand gesture control for
PowerPoint presentations provides a more natural and intuitive way of controlling the
content. These technologies have the potential to improve user engagement, interaction, and
overall experience, making teaching and learning more effective and enjoyable. However,
there is a need for further research and development to optimize these technologies and
identify their full potential. Overall, the use of virtual whiteboards and hand gesture control
for PowerPoint presentations represents an exciting new frontier in education and
communication technology.

Future Scope
Enhanced Gesture Recognition:

Continued advancements in computer vision and machine learning algorithms could lead to
more accurate and reliable hand gesture recognition systems. This could involve better
detection of subtle movements and gestures, as well as improved robustness in different
lighting conditions and environments.

Gesture-Based Interaction in Virtual Reality (VR) and Augmented Reality (AR):

Integrating hand gesture recognition with VR and AR technologies could enable more
immersive and intuitive user experiences. Users could interact with virtual objects and
environments using natural hand gestures, enhancing gaming, training simulations, and other
applications.

Healthcare and Rehabilitation:

38
Gesture recognition systems could be applied in healthcare settings for rehabilitation
exercises, remote patient monitoring, and surgical assistance. Combining gestures with voice
commands could allow healthcare professionals to interact with medical devices and access
patient information more efficiently.

Accessibility and Inclusion:

Improvements in gesture recognition and voice assistant technologies could further enhance
accessibility for individuals with disabilities, allowing them to interact with digital devices
and services more easily and independently.

Security and Surveillance:

Gesture recognition can enhance security and surveillance systems by enabling more
sophisticated monitoring and control mechanisms. For instance, systems can analyze
suspicious gestures or movements for threat detection, access control, and identification
purposes.

References and Bibliography:

1. https://www.geeksforgeeks.org/machine-learning/
2. https://amzn.eu/d/bfMQE6Y
3. https://developers.google.com/mediapipe/solutions/vision/gesture_recognizer#get_sta
rted

39
4. Hands-On Machine Learning with Scikit-Learn, Keras, and
TensorFlow by Geron Aurelien

40