GraphRAG-Based Agent Architecture for
Multimodal Knowledge Retrieval
Minor Project Synopsis
B.Tech Computer Engineering
7th Semester
Minor Project
CEN-792
Project Proposal By:
22BCS009 - Arib Ansari
22BCS040 - Harshit K. Gupta
Department of Computer Engineering
Faculty of Engineering & Technology
Jamia Millia Islamia
New Delhi (110025)
(Year-2025)
Abstract
The exponential growth of multimodal data—text, images, audio—poses major
challenges for traditional information retrieval systems. While
Retrieval-Augmented Generation (RAG) has improved large language models
by integrating external textual knowledge, it remains mostly limited to
text-only inputs and outputs. As a result, valuable insights embedded in
non-textual formats often go untapped, limiting the ability of AI assistants to
provide rich, context-aware responses across different media.
This project presents a GraphRAG-based Multimodal Knowledge Retrieval
Agent designed to address these limitations through unified processing,
storage, and retrieval of multimodal information. The system features a
multimodal ingestion pipeline that converts text, images, and audio into
semantically meaningful components. Text is tokenized and indexed, images
are analyzed using vision-language models to extract objects and
relationships, and audio is transcribed and annotated with acoustic events. All
extracted entities and their relationships are stored as nodes and edges in a
dynamic knowledge graph.
Using GraphRAG’s hierarchical community detection, the agent organizes
related entities into nested clusters with summary nodes at varying levels of
abstraction. At query time, it performs a two-stage retrieval: first identifying
relevant summaries through a global graph search, then exploring related
entities via local k-hop traversal. This hybrid retrieval approach enables fast,
scalable access to both high-level context and detailed evidence—while
preserving modality-specific insights and data provenance. The retrieved
multimodal context is then synthesized by a generator model to produce
cohesive, evidence-backed answers that can directly reference source images
or audio clips. This allows the agent to answer complex, cross-modal
queries that were previously intractable, paving the way for a new
generation of AI that can reason across a rich tapestry of interconnected
knowledge.
Introduction
Background
In the rapidly evolving landscape of artificial intelligence and information
retrieval, Retrieval-Augmented Generation (RAG) systems have significantly
enhanced the capabilities of large language models (LLMs) by integrating
external knowledge sources [12][32]. However, traditional RAG methods are
predominantly confined to textual data and rely on vector similarity search,
which limits their ability to capture deeper semantic relationships or handle
queries that span across different media types such as images, audio, and
diagrams.
The emergence of GraphRAG marks a major advancement by introducing
structured, graph-based knowledge representations and hierarchical
community detection to organize information more effectively [2][12]. Unlike
conventional RAG models, GraphRAG builds knowledge graphs from raw
data, extracts entity-level relationships, and supports multi-hop reasoning
through both global and local graph traversals. This enables richer,
context-aware retrieval and better support for complex queries.
Despite these innovations, existing GraphRAG implementations remain
focused on text. In an increasingly multimodal world—where learning
materials, notes, and knowledge sources often combine text with visual
diagrams, handwritten equations, and even audio explanations—this is a
major limitation. AI agents that fail to account for modality-specific signals risk
losing critical context, thereby underperforming in scenarios that demand
integrated reasoning across multiple data types.
Problem Statement
Current RAG and GraphRAG architectures exhibit several limitations when
applied to multimodal or personalized retrieval tasks:
1. Modality Isolation: Inability to jointly reason across text, image, and
audio data leads to fragmented understanding.
2. Context Loss: Vector similarity alone fails to maintain relational context
across semantically related concepts.
3. Scalability Bottlenecks: Lack of structured indexing leads to
inefficiency in large and heterogeneous datasets.
4. Lack of Personalization: Existing systems do not support long-term,
user-specific memory across different modalities.
This project aims to develop a GraphRAG-powered Multimodal Knowledge
Retrieval Agent capable of unified processing, graph-structured storage, and
contextual retrieval across text, image, and audio modalities. The system is
designed to support personalized, explainable answers grounded in structured
memory and multi-hop reasoning.
Use Case: Optimal Application of GraphRAG for a Personal
Knowledge Base
The primary use case for evaluation involves the construction of a personal
knowledge base (PKB) from academic materials including handwritten notes,
book excerpts, annotated diagrams, and lecture audio transcriptions.
Traditional vector-based search methods often return surface-level matches
without understanding conceptual links across modalities. In contrast, our
system:
● Extracts key entities and relationships (e.g., “Maxwell’s equations”,
“illustrates”, “derived from”) from all input formats.
● Builds a unified knowledge graph, clustering related entities into
semantic communities using graph-based community detection.
● Generates community-level summaries and indexes all raw snippets
in a vector store.
● Supports two-stage retrieval: identifying relevant topic clusters
globally, and drilling down locally for fine-grained, provenance-backed
evidence.
For example, a query like “Second Law of Thermodynamics” will surface the
Thermodynamics cluster summary, pull in specific notes, derivation diagrams,
and referenced figures, and synthesize an answer that cites each source.
Proposed Method / Algorithm
1. Multimodal Input Processing Layer
The agent processes and semantically segments input across three data types:
● Text: Chunked into ~600-token segments with overlapping context.
● Images: Processed using vision-language models for object detection,
captioning, and OCR.
● Audio: Transcribed via WhisperX, capturing text and temporal markers for
sequence-aware retrieval.
2. Knowledge Graph Construction
A dynamic graph is created using:
● Entity Extraction across modalities (e.g., text mentions, image objects, audio
speakers/events).
● Cross-modal Relationship Mapping, such as linking a captioned image
region with its text reference.
● Neo4j is used to store the graph, enabling efficient traversal and query.
3. Hierarchical Community Detection
Using the Leiden algorithm, the graph is organized into:
● Nested semantic communities, grouping related multimodal entities.
● Community Summaries generated at multiple abstraction levels to enable
high-level navigation.
4. Hybrid Retrieval Strategy
Queries follow a two-stage retrieval:
● Global Search over community summaries (via map-reduce).
● Local k-hop Search in the graph to collect detailed context.
● Qdrant Vector Store supports fallback retrieval via semantic embeddings for
novel or sparse queries.
5. Response Generation with Provenance
The retrieved multimodal context is synthesized via:
● LLMs (e.g., GPT-4o) to generate fluent, evidence-backed responses.
● Source Attribution, where answers are linked back to original text snippets,
image regions, or audio timestamps.
Programming Environment & Tools Used
Category Tools & Technologies
Core Languages Python 3.10
JavaScript (ES6)
Backend Frameworks & Libraries FastAPI (API layer)
Pydantic (data validation)
HTTPX (async HTTP client)
Multimodal Processing WhisperX (speech-to-text transcription)
CLIP (vision-language embeddings via LLM API)
Tesseract OCR (pytesseract)
Knowledge Graph & Storage Neo4j 5.x (graph database)
neo4j-driver (Python client)
Qdrant (vector store)
qdrant-client (SDK)
Hierarchical Clustering & leidenalg with igraph (community detection)
Summaries LLM API (community summary generation)
Retrieval & LLM Integration LangChain (prompt templates & pipelines)
LLM API (global/local search orchestration)
GraphRAG Toolkit (Microsoft's framework for
graph-enhanced RAG workflows)
Development Environment & Visual Studio Code (IDE)
Version Control Git & GitHub (source control, code review)
Deployment & Execution Python venv (virtual environments via conda)
Environment Windows 11 (OS)
External APIs & Services LLM API providers (e.g., OpenAI, Gemini, LLaMA
endpoints)
WhisperX Model Hub (audio transcription)
Hugging Face Transformers (pre-trained models)
Hardware CUDA enabled RTX 4060 (8GB).
References
1. Edge D., Trinh H., Cheng N., Bradley J., Chao A., Mody A., Truitt S.,
Metropolitansky D., Ness R. O., Larson J.
“From Local to Global: A GraphRAG Approach to Query-Focused
Summarization,” arXiv Preprint, arXiv:2404.16130v2, pp. 1–17, 2025.
2. Microsoft Research Team,
“GraphRAG Documentation,” Microsoft Open-Source Documentation Portal,
Version 1.4, pp. 1–35, 2025.
3. Lee J., Wang Y., Li J., Zhang M.
“Multimodal Reasoning with Multimodal Knowledge Graph,” Proceedings of
the 62nd Annual Meeting of the Association for Computational Linguistics
(ACL), pp. 579–590, 2024.
4. Paranyushkin D.
“Portable GraphRAG: Optimize Your LLM RAG with Knowledge Graphs,”
InfraNodus API Whitepaper, Vol. 2, No. 1, pp. 5–14, 2024.
5. Lee J., Wang Y., Li J., Zhang M.
“Multimodal Reasoning with Multimodal Knowledge Graph,” arXiv Preprint,
arXiv:2406.02030v2, pp. 1–20, 2024.
6. Microsoft GraphRAG Team,
“GraphRAG API Overview,” InfraNodus, API Documentation, pp. 1–16, 2025.
7. Liang J. et al.,
“LangChain: A Framework for Developing LLM Applications,” 2024.
8. Radford A. et al.,
“Learning Transferable Visual Models from Natural Language Supervision,”
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 8748–8763, 2021.
9. Xu Y. et al.,
“WhisperX: Enhancing Whisper Transcription via Forced Alignment,” arXiv
Preprint, arXiv:2306.13316, pp. 1–15, 2023.
