10 VII July 2022

https://doi.org/10.22214/ijraset.2022.46007
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue VII July 2022- Available at www.ijraset.com

Damage Detection and Classification of Road

Surfaces
Jeevitha D1, Himanshu Sharma2, Vivekanand S Gogi3
1, 2 3
Student, Assistant Professor, Industrial Engineering and Management, RV College of Engineering, Bengaluru, India

Abstract: Road damage refers to a condition where a road is not able to serve the traffic up to an optimal level. For
understanding the causes and the process of identifying road damages, a thorough literature survey of various international and
national journal, articles have been done and presented in the current study. The same thing has also been done to assess and
understand the current deep learning models used in road damage detection. Various common factors are identified that cause
road damage. The dataset is extracted from the AWS server which has a separate.tar.gz train and test dataset. The dataset is in
.jpg format which consists of images taken from three countries India, Japan, Czech Republic. The dataset of mixed images is
used to reduce the biases of the model and to increase its accuracy. The cleaning and analysis of the data are done where
different types of damages were classified and the categories having the least images are removed and not considered for further
analysis. Then the dimensions of the bounding box of train datasets are identified and the area is calculated to find the damages
which are taking more area. Two models are developed using MobilenetSSD and YOLOV5 and both are compared to find the
best model.
Keywords: Deep learning, Neural Network, Image Processing, Object Detection.

I. INTRODUCTION
Throughout the globe, roads are used by cars, buses, trucks, motorcycles, mopeds, pedestrians, animals, taxis, and other travelers.
Travel made possible by motor vehicles supports economic and social development in many countries. Yet each year, vehicles are
involved in crashes responsible for millions of deaths and injuries. Each year, 1.35 million people lost their lives on roadways
around the world. Every day, almost 3,700 people lost their lives globally in crashes involving cars, buses, motorcycles, bicycles,
trucks, or pedestrians. More than half of those killed are pedestrians, motorcyclists, or cyclists.
Roads are intended to make driving easier. However almost every vehicle also put a lot of pressure on the surface itself. This
consistent pressure can cause stresses to emerge in the road surface, which as a result causes cracking. Vehicle crash are estimated
to be the eighth leading cause of death globally for all age groups and the leading cause of death for children and young people 5–29
years of age.
The motivation for this project is to produce new and innovative solutions to the problems that have been recognized in the
company. Road infrastructure is considered one of the most crucial and valuable public assets because of its contribution to
economic and social development. It connects various businesses and provides access to various public and private sectors. There
are three ways by which the conditions of the roads are inspected: manual, semi-automated, and fully automated. Manual and semi-
automated methods are traditionally used, in the manual approach the inspection person walks or drives through the pavement to
inspect the condition of the road whereas in the semi-automated process road images are collected by a fast-moving vehicle, and
then the images are inspected by a person in the office.

II. LITERATURE REVIEW

The literature concerned with road damage caused by heavy commercial vehicles is reviewed. The main types of vehicle-generated
road damage are described and the methods that can be used to analyze them are presented. Attention is given to the principal
features of the response of road surfaces to vehicle loads and mathematical models that have been developed to predict road
response.
According to the authors Yachao Yuan, Yali Yuan, Thar Baker, Lutz Maria Kolbe, Dieter Hogrefe [1], Existing Road damage
detection systems mainly process data on clouds, however, they are not able to warn users timely due to the long latency. In this
paper, they propose FedRD: a novel privacy-preserving edge-cloud and Federated learning-based framework for intelligent
hazardous Road Damage detection and warning.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 4408
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue VII July 2022- Available at www.ijraset.com

The authors Taoran Wei, Danhua Cao, Caiyun Zheng, Qun Yang [2] propose a surface defect segmentation method based on defect
sample simulation, which only needs a few defects training samples. The simulation method requires one single defect reference
sample for training and can generate the same type of defect in the specified area.
According to the authors, Rodrigo Rill-García, Eva Dokladalova, Petr Dokládal [3] Recent Road crack detection methods obtain
appealing scores but typically allow a few pixels tolerance margin. This is acceptable for locating cracks, but not for measuring their
width. U-VGG19, obtains an F-score of 71.77% on CrackForest, which is superior to other approaches when no tolerance is
admitted.
According to the authors Rana Khallaf, Mohamed Khallaf [4], six application-based topics were identified: equipment tracking,
crack detection, construction work management, sewer assessment, 3D point cloud enhancement, and miscellaneous topics.
Analysis shows that deep learning has been beneficial in leveraging data in areas such as crack detection and segmentation of
infrastructure and sewers; equipment and worker detection and analysis and reporting on construction-related operations.
According to the authors, Jie Fang, Bo Qu, and Yuan Yuan [5], Visual-based Road crack detection becomes a hot research topic
over the last decade because of its huge application demands. Road crack detection is a special form of salient object detection task,
whose objects are small and distributed randomly in the image compared to the traditional ones, which increases the difficulty of
detecting.

III. OBJECTIVES
The primary objective of this paper is to create a model which can detect road damage and classify it into different categories. To
compare and calculate the efficiency of deep learning algorithms (Mobile Net SSD and YOLOV5). To identify various factors that
cause road damage. To replace traditional methods of identifying damages with efficient and reliable deep learning models. The
secondary objectives include Automation of Road Damage Detection using Deep Learning Object Detection Models. To conduct
market research to expand Aurigo’s domain. Using state of Art Object Detection Models and Computer Vision Algorithms, process
images of damaged roads can be used and detect various forms of Damage like Potholes, Ruts, Bumps, Wheel Marks, Pavement
Damage. Damage to Crosswalks, Lines, etc. Once Damage has been Identified using this Data, then inform relevant Stakeholders
(Like Transportation Departments / Road Agencies) so that they may plan maintenance/rebuilding activities.

IV. PROBLEM STATEMENT

The problem statement for this project may be stated as Classification and damage detection of road surfaces. Roads are an
important asset to any country and like one of the economic backbones of any country. Typical causes for the damages are the
materials used during construction, climate conditions, and traffic volume. damages need to be identified and patched as soon as
possible. The process starts from the identification of the damage to the patching of the damage. The manual inspection is very time-
consuming and can be biased. Therefore, there is a need to come up with a solution that can automate the inspection and
classification process.
Manual inspection is also a very expensive method that requires the use of a vehicle. There’s small damage or cracks on the road.
People can complain about it in the respective department, and this makes the process longer and more time-consuming. So, it can
be thought of as something where the person can take the image and upload it to the server of the responsible department and can
take action accordingly and the process can be integrated with the GIS (Geographical Information System) which can also provide
the location of the damage on the road.

V. METHODOLOGY
As the first step, there will be a studying of the various Existing methods that are used to detect road damage. Some of which
include visual observations by humans and quantitative analysis using expensive machines. Quantitative determination based on
large-scale inspection, such as using a mobile measurement system (MMS) (KOKUSAI KOGYO CO., 2016) or laser-scanning
method (Yu and Salari, 2011) is widely conducted. Image processing etc.
The next step is to collect data from various sources. The dataset will consist of damage categories, comprising cracks and potholes.
The next step is to generate the DL algorithm and model. python and deep learning models will be used. In validating step, help will
be taken from the technical team to check for bugs, errors, or any other improvements that can be done. There will also be a demo of
the model that will be conducted to validate the model.
The result is then generated by running the model. As a part of the model, object detection is used Object detection is a computer
vision technique for locating instances of objects in images or videos.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 4409
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue VII July 2022- Available at www.ijraset.com

Object detection algorithms typically leverage machine learning or deep learning to produce meaningful results. Given an image or
a video sequence, they localize instances of objects of interest. As in our case, they localize various damages on the road.
General Concepts: Evolved into an effective tool for analyzing big data, deep learning uses complex algorithms and artificial neural
networks to train machines / computers, learn from experience, and data / like the human brain. Classify and recognize images. In
deep learning, a convolutional neural network (CNN) is a type of artificial neural network commonly used for image / object
detection and classification.
The Artificial Neural Network (ANN) is a fully connected multi-layer neural network, as shown in the following figure. These
consist of an input layer, some hidden layers, and an output layer. All nodes in one layer are connected to all other nodes in the next
layer. By increasing the number of hidden layers, the network becomes deeper.
Python is dynamically typed, and garbage collected. It supports multiple programming paradigms, including structured
(especially procedural), object oriented, and functional programming. Due to its extensive standard library, it is often referred
to as the "battery-included" language. Programmers often fall in love with Python because of the productivity gains that Python
offers.
YOLO is an acronym for You Only Look Once. Version 5 is used. It was introduced by Ultralytics in June 2020 and is the most
advanced object identification algorithm currently available. This is a new convolutional neural network (CNN) that detects
objects with high accuracy in real time
Exploratory Data Analysis: In statistics, exploratory data analysis is an approach to analyze data sets to summarize their main
characteristics, often using statistical graphical approach and other data visualization methods. A statistical model can be used or
not, but primarily EDA is for seeing what the data can tell us beyond the formal modeling and thereby contrasts traditional
hypothesis testing.
Table 1: Concatenated Data
filename width height class xmin ymin xmax ymax
0 Czech_000539.jpg 3 600 D00 125 413 187 481
1 Czech_002830.jpg 3 600 D00 0 418 123 497
2 Czech_002385.jpg 3 600 D00 342 454 403 532
3 Czech_002385.jpg 3 600 D10 83 481 289 510
4 Czech_002385.jpg 3 600 D10 297 497 598 532

A. Comparison
The next step of the project is the comparison of the deep learning algorithms namely MobilenetSSD and YOLOV5. MobileNet is a
small, low-latency, and low-power convolutional feature extractor that can be built to perform classification, detection, or
segmentation like popular large-scale models, such as Inception SSD [20]. It is based on a depth-wise separable convolution. YOLO
is an acronym for 'You only look once', is an object detection algorithm that divides images into a grid system. Each cell in the grid
is responsible for detecting objects within itself. For the comparison of both the algorithms in this project, their performance is
evaluated based on three indices namely, precision, recall, and F1 score. Precision is defined as the percentage of correctly predicted
features (true positive) out of the total predictive features (true positive and false positive). Whereas the recall is the percentage of
correctly predicted features (true positive) out of the total features present in the class (true positive and false negative). Since both
precision and recall counter each other so to balance the F1 score is used.
F1 = 2 × (Precision × Recall) / (Precision + Recall). The goal of the comparison is to identify the algorithm, which is more efficient,
fast, reliable, and more accuracy.

Figure 1: Japan Image

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 4410
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue VII July 2022- Available at www.ijraset.com

B. Deployment
After training and validating the model the test phase comes where the model has been deployed. From the comparison phase, the
best model is identified and has been chosen for the deployment using pytorch. After executing the deployment part at the backend
an URL for the local system is generated at the end which can be opened in the browser. Then the user can upload the image from
their system and the output will be generated with the image containing a bounding box specifying the damage along with the
classification specifying the type of damage.
To display the data
df_czech = pd.read_csv('labels_Czech.csv')
df_czech.head()

D20 8381
D00 6592
D40 5627
D44 5057
D10 4446
D50 3581
D43 793
D01 179
D11 45
D0w0 1
Name: class, dtype: int64

Figure 2: Types of Damages

There can be seen 10 different classes of damage as mentioned in the beginning. The data is not balanced as class D43 (Cross Walk
Blur), D01 (Longitudinal Construction Joint Part Crack), D11 (Lateral Construction Joint Part Crack) and D0w0 are scarce whereas
D20 (Alligator Crack) is in abundance. So, entries with D43, D01, D11, and D0w0 can be dropped.
Plotting the area Graph

Figure 3: Area Distribution Graph

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 4411
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue VII July 2022- Available at www.ijraset.com

This plot tells us about the average area covered by the bounding box. D20 (Alligator Crack) has the highest mean area. D50 (Utility
Hole) has a smaller average area. This tells us that Alligator cracks are usually spread on more areas of the road followed by D44
(White line blur).

Figure 4: Area Distribution Plot

Most of the damage classes have a very small spread or variance. Though the green one, D20 (Alligator crack), has a widespread.
Widespread signifies that the damage is propagated more on the road rather than being concentrated at a single small point.

VI. RESULTS
Objectness is essentially a measure of the probability that an object exists in the proposed region of interest if the objectivity is high,
it means that the image window is likely to contain an object. As seen in the graph the objectness loss is decreasing with more test
runs and loss is less in validation than in the training model, which means the model is underfitting and moves towards optimal.
Classification loss gives an idea of how well the algorithm can predict the correct class of a given object.
As seen in the graph the classification loss is decreasing with more test runs and loss is less in validation than in the training model,
which means the model is underfitting and moves towards optimal.

Figure 5: Model Loss Graph

Model Loss: as seen in the graph with the increase in several epochs the model loss is decreasing. The optimal number of epochs
will be 7 as at that point both the train and test curve are intersecting each other.
GIoU: Generalized intersection over union maximize the overlap area of ground truth and predicts boundary box. It increases the
predicted box size by moving slowly towards the target box for non-overlapping cases. As seen in the graph the GIoU loss is
decreasing with more test runs and loss is less in validation than in the training model, which means the model is underfitting and
moves towards optimal.

Figure 6: Accuracy Comparison graph

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 4412
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue VII July 2022- Available at www.ijraset.com

From the above model accuracy comparison graph YOLO is better than MobileNet. Yolo is having an accuracy of 95% whereas the
Mobile net’s accuracy is 78%.
The results of the deployment platform in terms of Image classification and Identification are

Figure 7: Czech Result

IoT devices such as traffic camera footage, collected images / videos, are key data used to identify the presence of damage,
classify damage types, and gain deeper insights into road problems and imperfections. Can be used as. Various technologies used in
the project, such as Python, Deep Learning Models, PyTorch Framework, Streamlit, OpenCV, and Jupyter Notebooks, extend the
depth and scope of the project and make it easier to understand and apply. You can use the drone to collect video / image data for
this project and use it as basic data for various classifications and identifications. It's a fully automated cloud-based software
solution that allows you to use drones to upload and process video / image data and identify road damage. The drone's
data is geographically tagged to help authorities be informed of the exact location and nature of the damage and to plan and
schedule maintenance activities.

Figure 8: Platform opening image

VII. CONCLUSION AND FUTURE SCOPE

The solution is a cost-cutting initiative that government agencies can use to schedule maintenance and other related activities
through automated cloud software. This is a great benefit to the customer (transportation/department / institution) as it reduces the
manual and time-consuming data collection process. This has a significant impact on the public as it helps governments plan
and maintain road infrastructure through concrete data-driven strategies. The developed platform can be a standalone
application for users to upload images / videos, a front end that allows users to upload images / videos and return results based on
the images and defects present on the road. In the future, the same prototype can be extended to propose a single standardized model
that can be applied worldwide, or at least in many countries with the same road conditions. In addition, this work can be used as
a baseline and the experiment can be repeated by collecting more images from different countries including India, Japan and Czech
in different seasonal conditions, making each damage class more appropriate. It expresses and improves the robustness of the entire
image detection system. All damage categories. Use a variety of algorithms to develop models that improve the accuracy of
damage detection.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 4413
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue VII July 2022- Available at www.ijraset.com

REFERENCE
[1] Yachao Yuan, Yali Yuan, Thar Baker, Lutz Maria Kolbe, Dieter Hogrefe FedRD: Privacy-preserving adaptive Federated learning framework for intelligent
hazardous Road Damage detection and warning Elsevier 2021
[2] Taoran Wei, Danhua Cao, Caiyun Zheng, Qun Yang A simulation-based few samples learning method for surface defect segmentation Elsevier 2020
[3] Rodrigo Rill-García, Eva Dokladalova, Petr Dokládal Pixel-accurate Road crack detection in presence of inaccurate annotations Elsevier 2022
[4] Rana Khallaf, Mohamed Khallaf Classification, and analysis of deep learning applications in construction: A systematic literature review. Elsevier 2021
[5] Je Fang, Bo Qu, Yuan Yuan Distribution equalization learning mechanism for road crack detection Elsevier 2019

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 4414