FACULTY OF SCIENCE AND TECHNOLOGY

INTERNATIONAL UNIVERSITY OF EAST AFRICA

"AI-POWERED COMMUNICATION PLATFORM: ENABLING REAL-TIME

CONVERSATION BETWEEN DEAF AND HEARING INDIVIDUALS”

BY

GROUP MEMBERS REGISTRATION NUMBERS

ROBEL HAGOS MAHRAY 22/1092/BSCS-S
YOEL KESETE HADGEMBES 22/398/BSCS-M
HAMSE MOHAMOUD ABDIRAHMAN 22/507/BSCS-M
MACHUOR PETER RIAK 23-1653-BSCS-J
SELU FUNDI HENOCK 21/734/BIT-J

A proposal submitted to the school of computing and informatics in partial fulfilment for the
award of the degree of Bachelor of Science in computer science of the International University
of East Africa (IUEA)

[March, 2025]
Declaration
we, declare that this research proposal is my original work and has not been submitted for any
other degree award at any other university.

Signature: ___________________________
Date: _______________________________

Supervisor’s Approval:
This proposal has been submitted for review with the approval of the following supervisor:

[Supervisor’s Name]
[Supervisor’s Title, e.g., Lecturer, School of Computing and Informatics]
Signature: ___________________________

Date: _______________________________
Dedication

This research project is dedicated to:

 Our family, for their unwavering love, support, and encouragement throughout our
academic journey.

 All those who believe in the power of technology to make the world a better place.
Acknowledgement

We would like to express our sincere gratitude to all those who contributed to the successful
completion of this research project. First and foremost, I thank International Univerisity of East
Africa for providing the platform and resources necessary to undertake this study. Special thanks
go to my supervisor, Mr. Burnabus Asingye for their invaluable guidance, constructive feedback,
and unwavering support throughout the research process.
Table of Contents
Dedication........................................................................................................................................3

Acknowledgement...........................................................................................................................4

AI-Powered Communication Platform for Deaf and Hearing Individuals......................................1

Research Proposal........................................................................................................................1

1.0 Definition of Proposal........................................................................................................1

2.0 Components of a Research Proposal.................................................................................1

2.1 Title....................................................................................................................................1

2.2 Background / Introduction.................................................................................................1

2.3 Research Problem / Statement of the Problem..................................................................1

2.4 General Objective / Aim / Purpose....................................................................................2

2.5 Specific Objectives............................................................................................................2

2.6 Significance, Importance, Contribution.............................................................................2

2.7 Justification / Rationale.....................................................................................................3

2.8 Theoretical / Conceptual Framework................................................................................3

2.9 Scope..................................................................................................................................4

2.10 Literature Review............................................................................................................4

2.11 Hypothesis / Postulates / Research Questions.................................................................5

2.12 Methodology....................................................................................................................5

2.13 Ethical Considerations.....................................................................................................6

2.14 Environmental Considerations.........................................................................................6

2.15 Timeline...........................................................................................................................7

2.16 Limitations / Anticipated Problems.................................................................................7

2.17 References........................................................................................................................7

2.18 Appendices......................................................................................................................8
AI-Powered Communication Platform for Deaf and Hearing
Individuals

Research Proposal
1.0 Definition of Proposal

This document proposes the development of an AI-powered communication platform that

bridges the gap between deaf and hearing individuals through real-time sign language-to-text and
speech-to-text conversion. This project aims to facilitate natural conversation between these two
communities without the need for human interpreters.

2.0 Components of a Research Proposal

2.1 Title

AI-Powered Communication Platform: Enabling Real-Time Conversation between Deaf and

Hearing Individuals

2.2 Background / Introduction

Communication between deaf and hearing individuals often requires a human interpreter, which
can be costly, inconvenient, and unavailable in many situations. Despite technological
advancements, there remains a significant gap in accessible, real-time communication tools that
can bridge this divide effectively.

Sign language is the primary mode of communication for many deaf individuals. However, most
hearing people are not proficient in sign language, creating a fundamental communication
barrier. While some solutions exist, such as text-based communication or professional
interpretation services, these approaches have limitations in terms of cost, availability, and the
natural flow of conversation.

Recent advancements in computer vision, machine learning, and speech recognition technologies
have created new possibilities for developing more effective communication tools. These
technologies can now recognize gestures and transcribe speech with increasing accuracy, making
it possible to create a system that can translate between sign language and spoken language in
real-time.

2.3 Research Problem / Statement of the Problem

The primary problem this research addresses is the communication barrier between deaf and
hearing individuals in real-time interaction scenarios. Specific challenges include:

1
 1. Limited availability and high cost of human sign language interpreters
2. Lack of accessible technology solutions that enable natural, real-time conversations
3. Social isolation experienced by deaf individuals in predominantly hearing environments
4. Communication difficulties in emergency situations where immediate understanding is
critical

2.4 General Objective / Aim / Purpose

To develop an AI-powered communication platform that enables real-time, bi-directional

conversation between deaf and hearing individuals by translating sign language to text and
speech to text.

2.5 Specific Objectives

1. Design and implement a gesture recognition system that can accurately interpret basic
sign language gestures and convert them to text
2. Develop a speech recognition component that can transcribe spoken language into text
with high accuracy in various acoustic environments
3. Create a user-friendly web interface that facilitates seamless communication between
users
4. Implement real-time data transmission between devices with minimal latency
5. Evaluate the system's performance, usability, and effectiveness through testing with both
deaf and hearing users

2.6 Significance, Importance, Contribution

This project contributes significantly to:

1. Accessibility: Provides a tool that reduces communication barriers for deaf individuals in
daily interactions
2. Inclusion: Promotes greater social inclusion by enabling more natural conversation
between deaf and hearing communities
3. Independence: Reduces reliance on human interpreters, giving deaf individuals more
autonomy in communication
4. Emergency Response: Offers a potential solution for critical communication in
emergency situations
5. Technology Advancement: Contributes to the advancement of accessible technology
through practical application of AI, computer vision, and speech recognition
6. Education: Creates opportunities for improved educational experiences for deaf students
in mainstream settings

2
2.7 Justification / Rationale

According to the World Health Organization, approximately 466 million people worldwide have
disabling hearing loss, with this number projected to increase to 900 million by 2050. Despite
this significant population, technological solutions for communication remain limited and often
inadequate.

Existing solutions typically fall into three categories: text-based communication, professional
human interpreters, or rudimentary sign language translation apps with limited functionality.
Each has significant limitations:

 Text-based communication is slow and interrupts the natural flow of conversation

 Human interpreters are expensive and not always available
 Existing apps often have poor accuracy, limited vocabulary, or significant delays

The proposed system addresses these limitations by leveraging cutting-edge AI technologies to

create a more natural, accessible, and responsive communication tool. The timing is opportune
due to recent advancements in:

 Machine learning models for gesture recognition

 Improved speech recognition accuracy
 Faster processing capabilities of modern devices
 Widespread availability of devices with cameras and microphones

2.8 Theoretical / Conceptual Framework

This project is grounded in several theoretical frameworks:

1. Universal Design: The principle that products should be usable by all people without the
need for adaptation or specialized design
2. Computer Vision Theory: Particularly as it relates to gesture recognition and
interpretation
3. Natural Language Processing: Understanding how to process and interpret human
language
4. Human-Computer Interaction: Principles that guide the development of intuitive, user-
friendly interfaces
5. Assistive Technology Models: Frameworks for evaluating the effectiveness of
technologies designed to assist people with disabilities

The conceptual framework for this project involves:

 Input systems (video capture for sign language, audio capture for speech)
 Processing systems (gesture recognition, speech recognition)
 Output systems (text display)
 Communication infrastructure (real-time data transmission)

3
2.9 Scope

The project scope includes:

1. Development of a web-based application accessible via standard browsers

2. Implementation of gesture recognition for a limited vocabulary of common American
Sign Language (ASL) signs and phrases
3. Integration of speech-to-text functionality for English language
4. Real-time data transmission between devices
5. Basic user interface for both deaf and hearing users
6. Testing and evaluation with a sample group of users

The project does not include:

1. Support for sign languages other than ASL (initially)

2. Mobile application development (initial phase)
3. Avatar-based sign language visualization
4. Integration with third-party messaging platforms
5. Support for multiple languages (beyond English)

2.10 Literature Review

Sign Language Recognition Technologies Several studies have explored computer vision
approaches to sign language recognition. Bragg et al. (2019) reviewed the state of sign language
recognition and translation, highlighting challenges in capturing the complexity of sign
languages. Recent approaches using deep learning, particularly convolutional neural networks
(CNNs) and recurrent neural networks (RNNs), have shown promising results (Camgoz et al.,
2020).

Speech Recognition for Accessibility Speech recognition technologies have made significant
advancements in recent years. Muda et al. (2010) provided a comprehensive review of speech
recognition techniques, while more recent work by Graves and Jaitly (2014) demonstrated the
effectiveness of deep neural networks for speech recognition tasks. Google's Speech API and
OpenAI's Whisper have set new benchmarks for accuracy in diverse acoustic environments.

Real-Time Communication Systems WebRTC has emerged as a standard for real-time

communication on the web. Johnston and Burnett (2012) outlined the architecture and
capabilities of WebRTC, demonstrating its suitability for applications requiring low-latency
communication.

Assistive Technologies for Deaf Individuals Previous work on assistive technologies for deaf
individuals includes captioning systems (Kawas et al., 2016), visual alerts (Matthews et al.,
2017), and sign language translation apps. However, as noted by Bragg et al. (2021), many
existing solutions have limited functionality and poor user experience.

4
User Interface Design for Accessibility Principles for designing accessible interfaces have been
well-documented. W3C's Web Content Accessibility Guidelines provide standards for accessible
web design, while specific research on interfaces for deaf users by Bottoni et al. (2011) offers
insights on visual design preferences and interaction patterns.

2.11 Hypothesis / Postulates / Research Questions

Main Hypothesis: An AI-powered platform using computer vision and speech recognition can
effectively facilitate real-time communication between deaf and hearing individuals with
sufficient accuracy and minimal latency to support natural conversation.

Research Questions:

1. What level of accuracy can be achieved in recognizing and translating a core vocabulary
of ASL signs using TensorFlow.js and MediaPipe?
2. What is the minimum acceptable accuracy threshold for users to find the system
beneficial?
3. How does latency in translation affect the perceived quality of conversation?
4. What interface design best supports the cognitive load of simultaneous communication
and technology use?
5. How does the use of such technology affect the social dynamics of deaf-hearing
interaction?

2.12 Methodology

System Development:

1. Requirements Analysis and Design

o Conduct user interviews with both deaf and hearing individuals
o Define functional and non-functional requirements
o Design system architecture and user interface mockups

2. Development of Gesture Recognition Component

o Data collection and preprocessing of ASL gestures
o Model training using TensorFlow.js
o Implementation of gesture detection using MediaPipe
o Testing and refinement of gesture recognition accuracy

3. Development of Speech Recognition Component

o Integration of Google Web Speech API
o Implementation of real-time transcription
o Testing in various acoustic environments

4. User Interface Development

o Implementation of responsive web interface using React.js

5
 o Design of simultaneous display mechanisms for both users

5. Integration and Real-Time Communication

o Implementation of WebSocket/WebRTC for real-time data transmission
o System integration testing
o Performance optimization

Evaluation:

1. Technical Evaluation
o Accuracy measurement of gesture recognition
o Accuracy measurement of speech recognition
o Latency measurement in various network conditions
o System reliability testing

2. User Evaluation
o Usability testing with deaf participants
o Usability testing with hearing participants
o Evaluation of conversation quality and effectiveness
o Collection of qualitative feedback

Data Analysis:

 Statistical analysis of accuracy and latency metrics

 Thematic analysis of user feedback
 Identification of system limitations and potential improvements

2.13 Ethical Considerations

This project involves human participants and must adhere to ethical standards:

1. Informed Consent: All participants in testing will provide informed consent

2. Privacy and Data Protection: User data will be anonymized and secured
3. Accessibility: The research process itself will be accessible to deaf participants
4. Cultural Sensitivity: Recognition of sign language as a cultural and linguistic system
5. Expectations Management: Clear communication about the system's capabilities and
limitations
6. Inclusivity: Ensuring diverse representation in testing and feedback

2.14 Environmental Considerations

The environmental impact of this project is minimal, but considerations include:

1. Energy Consumption: Optimization of algorithms to reduce computational load and

energy use

6
 2. Digital Sustainability: Design for longevity and compatibility with existing devices
3. Accessibility of Hardware: Ensuring the system works on standard devices to prevent
unnecessary hardware consumption

2.15 Timeline

The project will be completed over a period of 9 months:

1. Months 1-2: Requirements analysis, system design, and literature review

2. Months 3-4: Development of gesture recognition component
3. Month 5: Development of speech recognition component
4. Months 6-7: User interface development and system integration
5. Month 8: Testing and evaluation
6. Month 9: Refinement, documentation, and presentation

2.16 Limitations / Anticipated Problems

1. Accuracy Limitations: Sign language recognition may not achieve perfect accuracy
2. Vocabulary Constraints: Initial implementation will support a limited vocabulary
3. Dialect and Regional Variations: ASL has regional variations that may affect
recognition
4. Environmental Factors: Lighting conditions and background noise may impact system
performance
5. Technical Requirements: Users need devices with cameras, microphones, and internet
connection
6. Real-Time Challenges: Network conditions may affect the speed of communication
7. Cultural Nuances: Some aspects of deaf communication may not translate effectively to
text

2.17 References

(Note: The following references are illustrative and should be replaced with actual citations from
your literature review)

1. Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., ... & Ringel
Morris, M. (2019). Sign language recognition, generation, and translation: An
interdisciplinary perspective. The 21st International ACM SIGACCESS Conference on
Computers and Accessibility, 16-31.
2. Camgoz, N. C., Hadfield, S., Koller, O., Ney, H., & Bowden, R. (2018). Neural sign
language translation. Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 7784-7793.
3. Graves, A., & Jaitly, N. (2014). Towards end-to-end speech recognition with recurrent
neural networks. International Conference on Machine Learning, 1764-1772.
4. Johnston, A. B., & Burnett, D. C. (2012). WebRTC: APIs and RTCWEB protocols of the
HTML5 real-time web. Digital Codex LLC.

7
 5. Muda, L., Begam, M., & Elamvazuthi, I. (2010). Voice recognition algorithms using Mel
Frequency Cepstral Coefficient (MFCC) and Dynamic Time Warping (DTW) techniques.
Journal of Computing, 2(3), 138-143.

2.18 Appendices
(i) Budget

 Development hardware (if required): $1,000

 Software licenses and services: $500
 User testing compensation: $1,000
 Miscellaneous expenses: $500
 Total: $3,000

(ii) Project Team

 Lead Developer: [Your Name]

 Academic Supervisor: [Supervisor's Name]
 Advisory Committee: [List committee members if applicable]

(iii) Instruments and Tools

Development Tools:

 TensorFlow.js for machine learning model development

 MediaPipe for hand tracking and gesture recognition
 Google Web Speech API for speech recognition
 React.js for front-end development
 WebSocket/WebRTC for real-time communication
 GitHub for version control
 VS Code for code editing

Testing Instruments:

 System Usability Scale (SUS) questionnaire

 Custom evaluation metrics for accuracy and speed
 Interview guides for qualitative feedback
 Performance testing tools for latency measurement

Hardware Requirements:

 Computers with webcams for development and testing

 Various devices for compatibility testing (laptops, tablets)
 Stable internet connection for real-time communication testing

8
9