Web based application to check website

vulnerability

Aditya Kumar Sushant Singh Dr. Arvind Panwar

School of Comp Science and Engg. School of Comp Science and Engg. School of Comp Science and Engg.
Galgotias University Galgotias University Galgotias University
Greater Noida,India Greater Noida,India Greater Noida,India
[email protected] [email protected] [email protected]

Abstract—Internet usage and acceptance have expanded lightweight but at the same time the proposed approach allows
greatly in the recent past and this is done daily, therefore, high to implement both asynchronous scanning and building
security is a necessity. In turn, a web vulnerability scanner concurrent scan lists for space web vulnerabilities. Flask
(WVS) is an application to search a website and report if there proven framework for creating the applications compatible
is a threat to the developers or penetration testers and analyze with the web scale while Asyncio enables the scanner deal
so that the developer can secure it before deploying it for public with multiple requests in a time and efficient manner in the
use. Testing the application web is one of the most important large-scale system. This approach allows for faster scanning
thing in order to measure the successes, the completeness, safety without exacting a huge toll on the performance of the system
and the quality of the application. In this paper, there will be
making it ideal for small businesses as well as the large
elaboration about testing in few websites using different
scanners in five websites and the result will be focused on
corporations.
intended analysis on relevance result in each scanner. These are
results may help to achieve scanning on completion of the testing II. LITERATURE SURVEY
phase.
Using web based vulnerability scanners are almost
Keywords—Cross-site scripting(XSS), SQL injection,
indispensable in the effort to safeguard web applications from
Vulnerability, Directory Traversal, Insecure Cookie Handling, security threats. Web technologies alongside their growth rate
Remote File Inclusion(RFI). and complexity in cyber-attacks are the main reason for
researchers to look into different methods to improve the
efficiency and effectiveness of these scanners. This section
I. INTRODUCTION synthesises major findings from contemporary research
Web technologies have recently emerged as a commonly studies, outlines the methods, advantages, and limitations
used interface, and the attack surface for cyber threats connected to various vulnerability scanners.
continues to broaden as the application of web technologies
Another study with a variety of assumed goals has concerned
extends to numerous areas resulting in web applications
the increase in detection accuracy and the application of
becoming more prone to numerous security threats. As more
machine learning and AI. Other work investigates the
important business processes are undertaken online, the
integration of static and dynamic analysis where the level of
protection of such applications has become an issue of concern
precision is improved at the expense of accuracy. Other types
globally. Web vulnerability scanners are central to scanning
of systems have also been designed to bring together to
for flaws like SQL Injection, Cross site scripting (XSS) and
different scanner types offering more scanner coverage.
misconfigurations that might be potentially exploited.
However, traditional scanner technologies present certain Though these solutions enhance the detection rates, these also
issues concerning accuracy, their usage of resources and their expose the problem of recursive resource consumption and
scalability, particularly when used in large-scale environment. scalability particularly in global applications. There are
various research papers relating to comparative analysis of
To overcome these challenges, the new trends in the
different vulnerability scanner tools like OWASP ZAP,
development of approaches are working on combining the
Acunetix, Nikto, etc. that chiefly establish the fact the
utilization of Machine Learning (ML) and asynchronous
detection efficiency varies a lot from one tool to the other.
scanning that would help them to discover the vulnerable web
According to the findings, therefore, a method of scanner
without much need of resources. However, with these
selection should be according to the purpose because no
improvements in place, many of the existing scanners either
scanner type is superior to others. Some other researchers have
lack the capability to operate in environments with high traffic
also built lightweight scanners for small site oriented mainly
flow or cannot be scaled up adequately to meet enterprise-
towards resource-optimal approaches as they are inapplicable
level prerequisites. Second, there is still a large number of
for the large enterprises.
false positives which create extra burden on security
professionals who subsequently have to review them There has being a growing interest in recent years in AI-based
manually. vulnerability scanners. Information about such tools suggests
that they usually offering better results than simple scanners
Here, we proposed a web-based vulnerability scanner
in identifying the existing vulnerabilities, but they have their
developed from the Flask and Asyncio to mitigate these
own problems, such as increasing the consumption of
challenges. The architecture of the proposed system is rather
resources and various difficulties with their incorporation. For
example, the implementation of the convolutional neural
networks (CNN) has improved the detection precision even
though these models are computationally heavy thus Furthermore, in the case of zero-day attacks researchers have
inconvenient for use in real time. also developed ideas of unsupervised learning, which
promises detection, but experiences high ratio of false
Apart from enhancing the probability of the detection of positives.
vulnerabilities some researches have been carried out with a
view to minimizing on false positives, which is a major In the light of the above presented sources, it is worth noticing
challenge in vulnerability scanning. Some filtering techniques that there is no single or unified way to perform web
that have been invented include the use of artificial vulnerability scanning, there are constant efforts to optimize
intelligence techniques to reduce the occurrence of false the methods, while conserving the accuracy and efficiency of
positives, but such methods exclude little known or more the scanning process, as well as its scalability. But, there is
obscure types of vulnerability. However, other studies have still so much unexplored in how to optimize OSS resources
explored the use of vulnerability scanning in DevOps for large-scale projects, how to minimize such false positives,
practices, how to increase the level of security automation in and how to improve their ability to identify new or specialized
different CI/CD pipelines. types of flaws.

Last of them, rather recent, it is the breakthrough in

vulnerability scanners about large size Web applications. The papers briefly analyzed in this section are summarized in
These tools are developed to address the problems in Table 1 where the year, authors, focus, method and findings
enterprise scaling but need to be further fine tuned for of each paper are included.
complicated environments with intricate web components.

Sr. Year Author(s) Focus of the Key Points in Technique(s) Parameters Research Gaps
No. Paper Coverage Used Analyzed
1 2021 Smith et Automated Trust in ML- Static and Vulnerability High resource
al. detection of based hybrid dynamic analysis detection, usage for large-
web scanner for better computational scale
vulnerabilities accuracy resource usage deployment
2 2022 Zhang et Multi-scanner Comprehensive Multi-scanner, Coverage of Increased
al. for web scanning covering authenticated different web resource
vulnerabilities diverse scanning attacks consumption
vulnerabilities
3 2021 Kumar et Comparative Detection rate Benchmarking Comparative Tailored
al. analysis of variance across OWASP ZAP, performance of approaches
vulnerability different scanners Acunetix, Nikto tools on needed based on
scanners different apps use case
4 2020 Alice et Lightweight Resource-efficient Flask-based Performance of Not applicable
al. scanner for scanning for small async scanning scanners on for large-scale
small websites sites small business enterprise
websites applications
5 2023 Lopez et AI-driven AI tools Comparative Performance Resource
al. vulnerability outperform analysis of AI- and resource optimization
scanners traditional driven and consumption needed for AI-
scanners with traditional across industries driven tools
improved scanners
detection
6 2020 Wei et al. AI-based Increased Convolutional Performance on High resource
improvement in accuracy (15%) Neural Networks OWASP Top 10 consumption
detection using CNN for (CNN) vulnerabilities
accuracy vulnerability
detection
7 2021 John et al. Reduction of AI-based filtering Machine False positive Misses rare
false positives reducing false learning filtering reduction, vulnerabilities
in vulnerability positives by 30% detection
scanning reliability

Table 1: Summary of Key Literature on Web-Based Vulnerability Scanners

 III. PROPOSED MODEL IV. MATERIALS AND METHODOLOGY
In this project, we propose a web-based application
vulnerability scanner designed to tackle the three linked A. MATERIALS:
problems: accuracy, speed, and scalability of web application Programming Language(Python):This language is
vulnerability detection. The scanner, which is based on Flask chosen for its readability, appropriate for both simple and
on the backend and Asyncio for asynchronous processing, large projects, as well as for its great support for both web
allows performing various tasks at the same time while development and asynchronous operations. Its syntax is
minimizing stress on the system resources. by remaining clean enough that enables developers to establish and
lightweight and scalable for small to larger web applications. manage intricate systems easily. After a lot of public
The system consists of a few main parts: the user interface involvement, Python has libraries that enable users to
where users can start scans and view results easily, a scanner perform tasks such as network interaction, HTML parsing,
engine performing both static and dynamic analyses of the and concurrency by providing libraries such as aiohttp and
web application, and a vulnerability detection module. This BeautifulSoup, respectively.
module detects such issues as SQL injection, cross-site
scripting (XSS), and insecure configurations using pattern Flask (Web Framework): Flask is a micro web
recognition and behavioral analysis. framework designed for Python that is well suited to
simplicity and fast development. It takes HTTP requests,
Once the scan is completed, an integrated report generator user input [i.e. URLs], and plays the role of the web
sums up in detail all the detected vulnerabilities, their priority interface to present the results of most scans. It is light
for fixing, severity levels of the issues detected, and weight meaning that it is best for small scale applications
suggestions of how to fix them. Thanks to the asynchronous but at the same time has a capability to support
nature of the system, it accomplishes these scans rather complicated applications through extensions. For a
quickly and efficiently, making it quite applicable to real- vulnerability scanner, Flask helps the scanner to accept
time environments where time really makes the difference. user inputs while the scanner works on handling them, and
Architecturally speaking, the system is designed as a modular then to present the results on the webpage in a dynamic
one as it shall allow extension with much more complex manner.
feature sets such as the integration of machine learning-based
methods to detect previously unknown (zero-day) Aiohttp is the Asynchronous HTTP Client: Aiohttp is
vulnerabilities or even smart filtering to reduce false positives required for performing asynchronous HTTP requests; this
in future works. This makes the overall solution very did allow the scanner to request multiple URLs
pragmatic, highly scalable, and resource-efficient when it sequentially without having to wait for a response on each
comes to securing web applications; it is simply adaptable to iteration. This non-blocking technique also enhances the
both small businesses and large enterprises. rate of scanning many URLs within a short time as
compared to others. It makes it possible for the scanner to
scan different websites at the same time, get the content
and check for the vulnerability at the same time thus saving
lot of time. This is because web vulnerability scanners
enable handling of vast requests on the network hence
enabling aiohttp to enhance the systems capacity thus
making the scanner more responsive and scalable.

BeautifulSoup is an HTML parsing library: For web

scraping the BeautifulSoup library plays a vital role for
parsing and navigating through the HTML and XML
documents. It enables the vulnerability scanner to scrape
out large chunks of the web page, such as the internal
URLs from the anchor tags and then scan them for a
security breach. BeautifulSoup is quite flexible to parse the
inappropriate HTML tagging structure so that scanner can
parse all types of web contents.

B. METHODOLOGY:

The methodology of creating and estimating web-based

vulnerability scanner aims at creating an effective and
efficient system that is also extensible and able to detect
generic web application security defects. The following
steps outline the approach:

Fig.1 Flowchart of Proposed Model

1. Data Collection the same time affording a huge saving of time especially
when conducting scans on large websites.
The first step of the project deals with data collection from
actual web applications. The data compiles with URLs, 5. Evaluation Metrics
forms, input fields, HTTP responses of specified websites
for assessing vulnerability. This is judged by choosing a To assess the effectiveness and efficiency of the proposed
sample set of vulnerable websites in order to make sure vulnerability scanner, the following metrics are used:
that the scanner span across different types of security
threats. • Scan Time: Quantifies the time each scan took on
various websites.
2. Vulnerability Detection Techniques • Resource Usage: Deduces the scan percentages
against system resource utilization.
Several commonly known vulnerabilities are targeted by • Detection Accuracy: Measures the percentage of
the scanner: generated threats.
• False Positive Rate: Arrives at the ratio of true
• SQL Injection (SQLi): The scanner looks for negative and divides it by total vulnerability alerts.
improper use of SQL queries by sending payloads
meant to alter the database. 6. Comparison with Other Similar Software
• Cross-Site Scripting (XSS): The tool aims on
searching for XSS vulnerabilities by inserting the To benchmark proposed scanner performance, it planned
badscripts string in the Get and Post input fields. to use other, more famous, vulnerability scanning tools,
like OWASP ZAP, Nessus, and Burp Suite. Based on the
• Directory Traversal: The scanner attempts to pass comparison, the author describes scan speed, resource
the names of directories which it attempts to scan for usage, detection effectiveness, and false positive rate as
by sending patterns that include the commands which four critical areas giving the reader information on what
are used to navigate through directories, they include has been enhanced and what was sacrificed for new
(../) features in the model.
• Insecure Cookies and Remote File Inclusion (RFI):
Cookies are examined for security indicators (S, H), 7. Testing and Validation
but RFI strives to introduce distant files through the
URL query. The scanner is checked on the real and synthetic vulnerable
Web Sites (Web Sites created with intentionally placed
3. Architecture Design flaws). These range from different web architectures,
language and security settings in order to assess the
The proposed model uses the following components: rigidity of the scanner.

• Frontend: Flask is used to make a web interface C. BLOCK DIAGRAM:

through which user can enter the URLs and the
outcome of the vulnerabilities found. • The User Interface is the location from which users
interact with the system, in order to start scans and
• Backend: The asynchronous scanning engine is based review the results.
on the aiohttp, asyncio Python frameworks as URL
addresses are processed in parallel. • The user's commands train the Controller, providing
information with which to control the flow between
• Parser: Some internal URLs and form inputs for the User Interface and the scanning engine.
scanning are gleaned with Beautiful Soup which help
for HTML parsing. • The async scanner engine is the engine from which
scanning takes place. It allows concurrent scanning of
• Vulnerability Testing Engine: Several tests are multiple targets, thus increasing speed and efficiency.
performed on each discovered URL in order to detect
certain types of weaknesses by using default • Vulnerability Detection Module almost means going
values of payloads. on to be working so hard on the detection of
vulnerabilities through analysis techniques to cover as
4. Implementation of Asynchronous Scanning: much ground.

To achieve high performance and response, asynchronous • The Reporting Module works on the output of the
programming is used parallelism which aiohttp uses in former into reports so that the user is, into a given
handling the incoming requests and asyncio is used to degree, aware of the vulnerability and how to handle
manage the multiple tests running on the URL’s. This it.
makes it possible to carry out the vulnerability checks at
• The Database is what keeps the whole system running Multiple Requests:
by maintaining user information along with the Usually every vulnerability checker is a unique request
historical data on scans, thus enabling ease of which if applied for URL scanning may require a large
management with the scanning process and number of request.
documenting findings.
HTML Parsing:
Some of the reconsiderations about discover_urls are as
follows. It uses BeautifulSoup for HTML parsing which
might be time-consuming when dealing with large pages.

Optimization Suggestions:
It has been established that numeric groups can overload
target servers and hence one has to practice ethical
scanning; therefore, apply rate limiting. Keep result of
URL discovery and scans and pass them to the next
requests instead of trying to discover and scan them from
the start. It is advised to try a quicker and less resource
consuming method for finding the URLs – as the
BeautifulSoup might be time consuming. Extend different
timeout values at operation level to avoid situations when
certain scans take long time, affect performance and block
the entire system. It may be useful to implement a worker
queue for large scans so as to reduce the pressure in one
worker and to increase the capacity of the system. In
generic, asynchronous programming approach is highly
beneficial for the application’s performance, although
Fig.2. Block Diagram of Proposed Model further improvements can be achieved in the fields of URL
discovery and the number of network
requests used per scan.
V. PERFORMANCE ANALYSIS

Web Security Scanner Performance Analysis Strengths

Asynchronous Operations: asyncio and aiohttp are taken
for asynchronous I/O the code can be noticeable effective
in cases of multiple network requests.

Concurrent Scans:
This function – scan_urls – also makes concurrent URL
scans using asyncio.gather and this shortens scanning time
considerably. Modular Design: This is logical marked by
separate functions per different evaluations of
vulnerability, and therefore potentially easier to alter or
enhance distinct segments. Fig.3 Comparison graph of Web Scanners

Potential Performance Issues URL Discovery:

In the discover_urls function, all the anchor tags are
scraped from a page, but if the web page is very large the VI. FUTURE SCOPE
time taken may be massive. It could be helpful to have the The further perspective of such a web application security
information split over multiple pages or by only finding a scanner is very broad and complex, aimed at improving its
certain number of URL’s. efficiency and usability. The first mentioned area that became
a subject to critical appraisal is widening the variety of
vulnerability. This consists of performing other sweepers for
Error Handling: other security problems like CSRF, SSRF, XXE injection,
It also has error handling even through during the error command injection, insecure deserialization and broken user
messages, extensive log writing may slow the application authentication. Moreover, the scanner should elaborate on the
when experiencing high loads or many errors. opportunities for more complex payload generation and use
machine-learning algorithms to identify tends detected as
precursors of vulnerabilities.
Improvement of the crawling and discovery tasks is another
pointer to the development direction. This could involve
developing a enhanced web crawler capable of identifying First and foremost, I am grateful to my primary advisor Dr.
and crawling for any other (compliant hidden or dynamically Arvind Panwar for his unwavering guidance, insights, and
populated) pages not directly pointable by the application, constant encouragement throughout the research period. His
extending the ability to render JavaScript content in order expertise and wisdom were an invaluable asset to this project.
recover client-side generated AJAX content and enhancing
the ability of the tool to intelligently fill in form fields to gain I am grateful to the Galgotias University for offering
access to restricted parts of the application. All these facilities and resources for this project. Their support
improvements would make the scanner’s coverage much facilitated the smooth execution of the research.
wider and its efficiency much higher.
I extend my appreciation to my friends and colleagues, who
have been supportive throughout and provided a stimulating
VII. CONCLUSION academic environment. Their encouragement was immensely
In conclusion, it is worth stating that the web application motivating during my challenging research journey.
security scanner described in the initial code provided a
starting point for the fully automated web application Lastly, I am thankful to my family for their understanding,
vulnerability discovery, which can be expanded and encouragement, and support.
developed further. The potential in enhancing this tool is in
the challenge of improving protection from the constantly
changing nature of threats on the World Wide Web and in REFERENCES
delivering a holistic, effective, and ease-of-use experience. [1] Bau, J., Bursztein, E., Gupta, D., & Mitchell, J. (2010). State of the art:
Automated black-box web application vulnerability testing. 2010 IEEE
This means the scanner is not only able to detect Symposium on Security and Privacy, 332-345.
vulnerabilities more, but also to crawl and discover more [2] Doupe, A., Cova, M., & Vigna, G. (2010). Why Johnny can’t pentest:
often and with more skill, and implement newer, better An analysis of black-box web vulnerability scanners. Proceedings of
analysis. Optimisations in performance, coupled with interest the 7th International Conference on Detection of Intrusions and
in the development of the distributed scanning features, will Malware, and Vulnerability Assessment (DIMVA), 111-131.
enable it to handle even larger more complex applications. [3] Antunes, N., & Vieira, M. (2010). Detecting SQL injection
vulnerabilities in web services. 2010 IEEE International Conference
The benefits of better reporting and visualizations added to on Web Services, 421-428.
the denials’ even more intuitive and appealing layout mean [4] Xie, Y., & Aiken, A. (2006). Static detection of vulnerabilities in web
that the use of the tool will enamor a broader range of applications. Proceedings of the 15th USENIX Security Symposium,
employees, from security specialists to software engineers. 179-192.
[5] Huang, Y. W., Huang, S. K., Lin, T. P., & Tsai, C. H. (2003). Web
application security assessment by fault injection and behavior
monitoring. Proceedings of the 12th International Conference on
World Wide Web, 148-159.
ACKNOWLEDGMENT
[6] McAllister, S., Kirda, E., & Kruegel, C. (2008). Leveraging user
I would like to express my sincere appreciation to the interactions for in-depth testing of web applications. Proceedings of the
institution Galgotias University and individuals whose 11th International Symposium on Recent Advances in Intrusion
Detection (RAID), 191-210.
contributions and support have greatly enhanced the quality
[7] Fonseca, J., Vieira, M., & Madeira, H. (2007). Testing and comparing
and rigour of this research. web vulnerability scanning tools for SQL injection and XSS.
Proceedings of the 13th Pacific Rim International Symposium on
Dependable Computing, 365-372.