2021 International Conference on System Science and Engineering (ICSSE)

Enhancement of Robustness in Object Detection

Module for Advanced Driver Assistance Systems
Le-Anh Tran Truong-Dong Do
Dept. of Electronics Engineering Dept. of Artificial Intelligence Convergence
Myongji University Chonnam University
2021 International Conference on System Science and Engineering (ICSSE) | 978-1-6654-4848-2/21/$31.00 ©2021 IEEE | DOI: 10.1109/ICSSE52999.2021.9537944

Gyeonggi, South Korea Gwangju, South Korea

[email protected] [email protected]

Dong-Chul Park My-Ha Le

Dept. of Electronics Engineering Faculty of Electrical and Electrinics Engineering
Myongji University HCMC University of Technology and Education
Gyeonggi, South Korea Ho Chi Minh City, Vietnam
[email protected] [email protected]

Abstract — A unified system integrating a compact object Advanced Driver Assistance System
detector and a surrounding environmental condition classifier
for enhancing the robustness of object detection scheme in
advanced driver assistance systems (ADAS) is proposed in this
paper. ADAS are invented to improve traffic safety and
effectiveness in autonomous driving systems where object
detection plays an extremely important role. However, modern
object detectors integrated into ADAS are still unstable due to
high latency and the variation of the environmental contexts in
the deployment phase. Our system is proposed to address the
aforementioned problems. The proposed system includes two
main components: (1) a compact one-stage object detector
which is expected to be able to perform at a comparable
accuracy compared to state-of-the-art object detectors, and (2)
an environmental condition detector that helps to send a
Fig. 1. Advanced driver assistance systems (ADAS) [8].
warning signal to the cloud in case the self-driving car needs
human actions due to the significance of the situation. The
empirical results prove the reliability and the scalability of the traffic effectiveness, prevent traffic accidents, and facilitate
proposed system to realistic scenarios. fully autonomous driving in near future. However, modern
object detectors deployed in ADAS are still unstable due to
Keywords—ADAS, object detection, autonomous driving, deep many factors including how to select an appropriate network
learning, intelligent systems. architecture that can properly balance the speed-accuracy
trade-off. The inference speed is vastly important because
I. INTRODUCTION deployment to mobile devices has many stringent require-
Recent technological breakthroughs of convolution neural ments on latency and computational resources, while mobile
networks (CNNs) and the outstanding evolution of Graphics devices usually support limited built-in hardware resources.
Processing Units (GPUs) that can boost the performance of We are in need of reducing the latency by using relatively
parallel computation have made deep learning become the small networks while maintaining the accuracy as much as
dominating approach for various computer vision tasks. CNN- possible.
based object detection, particularly, is a computer technology
The goal of this paper is to enhance the robustness of the
that has attracted a huge party of researchers for the past
object detection module in ADAS by determining an object
decade because of its applicability. Generally, there exist two
detection network that is able to efficiently balance the trade-
genres of object detectors: one-stage object detectors which
off between inference speed and detection accuracy. In this
show high inference speed and considerable accuracy, the
paper, we propose a YOLO-based object detector which is
most popular one for this one-stage type is YOLO [1,2],
constructed based on YOLOv2 [1] with only 17 convolutional
because of the superiority in inference speed, one-stage object
layers in its backbone in order to achieve a low latency as well
detectors generally are integrated into many real-time object
as a favorable detection accuracy. We also find that a critical
detection systems and mobile devices; two-stage object
reason for the instability of object detectors during the
detectors, alternatively, show higher object recognition and
deployment phase is the variation of the environmental
localization accuracy but with more expensive computational
contexts. An object detector is trained with certain weather
cost and diminished speed such as Faster R-CNN [3].
scenes may not perform properly with other weather scenes
Autonomous driving vehicles, on the other hand, have that it is not trained to work with. To address this problem, we
been considered the future of technology as they have drawn further propose an environmental condition classifier and a
huge attention for the last decade. Many studies regarding communication protocol between the system and the cloud via
autonomous robots are conducted such as autonomous drones internet connection so that the system is able to send a warning
[4] and self-driving cars [5-7]. Advanced driver assistance
systems (ADAS), consequently, are introduced to improve

978-1-6654-4848-2/21/$31.00 ©2021 IEEE 158

Authorized licensed use limited to: National Taipei Univ. of Technology. Downloaded on May 06,2025 at 06:40:26 UTC from IEEE Xplore. Restrictions apply.
signal to the cloud in case the self-driving car needs human
actions due to the significance of the situation.
We first briefly introduce ADAS and their challenges in
Section II. Then, in Section III, we propose our system that
can be used for improving the robustness of the object
detection module in ADAS. The experiments are discussed in
Section IV. At last, conclusions of the paper and the future
works are provided in Section V.
II. ADAS
A. Brief Introduction to ADAS
Advanced driver assistance systems (ADAS) are
automated electronic systems which assist drivers in driving
and parking in order to increase car and road safety. ADAS Fig. 2. Example data with different light conditions in BDD100K dataset.
use a set of sensors and cameras such as ultrasonic, radar,
LiDAR, infrared and vision sensors, 2D camera, and 3D object detector of an autonomous car is trained in sunny
camera to detect nearby obstacles or driver errors and predict weather scenes, but in some moments, the weather condition
possible dangers on an earlier stage then respond to dangerous changes, then the detector will look at scenes that it is not
situations accordingly by analyzing the behavior of the trained to work with. In such cases, it may not perform at its
vehicles Due to the increasing need for mobility, traffic has optimal accuracy, or at different times in a day, the object
become more complex and therefore has become a greater detection performance can be different. Fig. 2 shows four
challenge for all vehicle users. Hence, ADAS are crucial in example data with different light conditions in BDD100K
order to avoid accidents and possible injuries or fatalities [8]. dataset, from top-left to bottom-right: daytime, twilight,
Though ADAS provide solutions for intelligent driving, they nighttime with street light, and nighttime without street light.
do not act autonomously but simply provide additional For a superb object detector which can perform well in
information about the traffic situation to support and assist different contexts, a neural network architecture with
drivers in implementing essential actions. The synchroniza- sophisticated and heavy computational cost is desired for great
tion of the actions of the driver and the information from the accuracy of the system. However, deploying such bulky
environment and furthermore the recognition of the current models on mobile devices for instant usage is undesirable
situation is extremely essential for obtaining the best of because deployment to mobile devices has numerous stringent
ADAS. For the last few years, various types of complex requirements on latency and computational resources. One
control units have been developed and integrated into ADAS way to solve this problem is to deploy the model to the cloud
[8]. An example of ADAS is illustrated in Fig. 1 which is and call it whenever its service is required. However, this
adapted from [8]. would require a high-quality internet connection which is not
B. Challenges and Proposals always available for moving objects like cars, thus it becomes
a constraint in production. This problem can be solved if it is
As mentioned earlier, ADAS gather information from the
implemented in a smarter way; that is, different models are
surrounding environment to support driving, and the object
trained and each one is able to handle a specific situation
detection plays an extremely crucial role in ADAS. Object
perfectly. We then only need to call an alternative model
detectors in ADAS currently have faced many challenges due
sometimes when there is a significant change in the
to the demands on inference speed and accuracy. While the
environmental condition. In this paper, we integrate an
inference speed depends mainly on hardware resources and
environmental condition classifier into our system. In order to
the complexity of the network, the accuracy, on the other
send a warning signal to the cloud whenever there is a
hand, depends completely on the detection algorithm adopted
significant environmental change, the system merely requires
for the system. In the scenario that the hardware technology is
an acceptable internet connection between the mobile device
developing rapidly, the future detection algorithms are
and the cloud.
expected to perform at higher speeds in near future. Currently,
the best choice for the object detector in a self-driving car is a III. ROBUSTNESS ENHANCEMENT
network which is able to balance efficiently the inference
speed and detection accuracy. In order to achieve this purpose, This section provides two techniques integrated into a
we propose a YOLO-based object detector that can ensure a single system that can be applied to ADAS to improve the
low latency as well as a favorable detection accuracy. robustness of the object detection module.

In the driving scenarios, object detection models popularly A. YOLO-based Object Detector
are trained and tested on driving datasets such as BDD100K The core of an object detection module definitely is an
[9] and Cityscapes [10]. However, those datasets include object detection network. Generally, there are two genres of
image data that are collected in certain areas, which means that object detectors: one-stage object detectors which show high
an object detector which is trained on mentioned datasets may inference speed and considerable accuracy, and two-stage
not perform as designed for when it is tested with another object detectors which yield higher object recognition and
dataset which contains image data gathered from another city localization accuracy but with expensive computational cost
with very different scene contexts. The problem can be seen and diminished speed [6]. Because of the superiority in speed,
from a simpler perspective when an autonomous car operates one-stage object detectors generally are integrated into real-
in many different weather conditions in a day or different light time object detection systems and mobile devices such as
conditions between daytime and nighttime; for instance, the mobile phones, autonomous vehicles, and drones. Since this

159
Authorized licensed use limited to: National Taipei Univ. of Technology. Downloaded on May 06,2025 at 06:40:26 UTC from IEEE Xplore. Restrictions apply.
 Fig. 4. Diagram of our system.

Fig. 3. Our proposed object detector (Backbone17-Det). (Backbone17). In our prior work, we have performed experi-
ments to verify the effectiveness of additional modules such
research is in the direction of enhancing the robustness of
as residual module [11], channel attention module (SE) [12],
object detectors in ADAS, one-stage detection scheme is
and spatial attention module (CBAM) [13] to choose the ones
chosen as the main type of network to investigate in this paper.
which are truly beneficial for improving our network
In one-stage object detectors, an input image is passed through
performance. The experiments have shown that the network
a backbone network to extract features and produce a final
with SE outperforms the one with CBAM in our case [6].
feature map with distilled information of objects, object
Therefore, we integrate residual connection and SE into our
recognition and localization of bounding boxes are performed
network backbone architecture. Our network (Backbone17-
directly on this feature map and done in a single pipeline
Det) is visualized in Fig. 3, the residual connection is applied
without any other post-processing steps.
once in every level of feature map resolution while SE is
In this study, we propose an efficient one-stage object applied before every downscaling step, specifically, SE is
detector based on our prior work [6] whose architecture was applied right after layer 5, layer 8, layer 13, and layer 17.
constructed based on the classical object detection algorithm
YOLO. Table 1 summarizes the backbone architectures of B. Environmental Condition Classifier
YOLOv2 [1] (Darknet-19) and our proposed object detector During onboard deployment, the object detector of an
autonomous driving car is expected to operate at a similar
TABLE I. DARKNET-19 AND BACKBONE17 ARCHITECTURES performance level as it is validated in training and testing
stages even under varying and complex environmental
Filters / Stride Output conditions, but this seems infeasible. In fact, there are multiple
Layers Types
Darknet-19 Backbone17 Resolution factors that an onboard object detector relies directly and
1 Conv 3 x 3 x 32 3 x 3 x 32 608 x 608
heavily on such as traffic density, road type, and time of the
day. Those factors impact directly the object detector and the
Maxpool 2x2/2 304 x 304
instability and the degradation of its performance without
2 Conv 3 x 3 x 64 3 x 3 x 64 / 2 304 x 304 warning is occasionally inevitable. This may lead the whole
Maxpool 2x2/2 152 x 152 system to implement unsafe and risky actions due to unreliable
3 Conv 3 x 3 x 128 3 x 3 x 128 / 2 152 x 152 object detection responses. The researchers in [14] introduce
4 Conv 1 x 1 x 64 1 x 1 x 64 152 x 152 a cascaded neural network that monitors the performance of
5 Conv 3 x 3 x 128 3 x 3 x 128 152 x 152 the object detector by predicting the quality of its mean
Maxpool 2x2/2 76 x 76
average precision (mAP) on a sliding window of the input
frames, the proposed cascaded network exploits the internal
6 Conv 3 x 3 x 256 3 x 3 x 256 / 2 76 x 76
features from the deep neural network of the object detector.
7 Conv 1 x 1 x 128 1 x 1 x 128 76 x 76
Similarly, but in a much simpler manner, we address this
8 Conv 3 x 3 x 256 3 x 3 x 256 76 x 76 problem by proposing an environmental condition classifier to
Maxpool 2x2/2 38 x 38 recognize and send a warning signal to the cloud whenever
9 Conv 3 x 3 x 512 3 x 3 x 512 / 2 38 x 38 there is a significant environmental change. Because this
10 Conv 1 x 1 x 256 1 x 1 x 256 38 x 38 system requires only an acceptable internet connection
11 Conv 3 x 3 x 512 3 x 3 x 512 38 x 38
between the mobile device and the cloud, this connection
protocol may help the authorities to take crucial actions from
12 Conv 1 x 1 x 256 1 x 1 x 256 38 x 38
the station instead of leaving the car handle by themselves in
13 Conv 3 x 3 x 512 3 x 3 x 512 38 x 38 the case the car needs human actions due to the significance
Maxpool 2x2/2 19 x 19 of the situation. The environmental condition classifier is
14 Conv 3 x 3 x 1024 3 x 3 x 1024 / 2 19 x 19 integrated along with the object detector in our system. Fig. 4
15 Conv 1 x 1 x 512 1 x 1 x 512 19 x 19 describes the diagram of our system.
16 Conv 3 x 3 x 1024 3 x 3 x 1024 19 x 19
IV. EXPERIMENTS
17 Conv 1 x 1 x 512 3 x 3 x 1024 19 x 19
18 Conv 3 x 3 x 1024 19 x 19
In this section, we first provide experimental constraints,
and the experimental results of the object detector as well as
19 Conv 1 x 1 x 1000 19 x 19
the environmental condition classifier are provided in order to

160
Authorized licensed use limited to: National Taipei Univ. of Technology. Downloaded on May 06,2025 at 06:40:26 UTC from IEEE Xplore. Restrictions apply.
Fig. 5. Person detection results using our object detector (Backbone17-Det), red boxes indicate the ground truth, green boxes depict detection results.

prove the applicability and the scalability of our system. We truck, motor, car, train, and rider. For the sake of simplicity,
also compare our result with other methods in terms of object we only concentrate on executing experiments and analysis on
detection efficiency. the person object category. The person data includes more
than 22K images (91K person objects) for training and about
A. Experimental Constraints 3.2K images (13K person objects) for testing. We train our
We propose the system in the scenario that we do not network with multi-scale training strategy; that is, the input
possess a real model of autonomous driving car. Therefore, size scales from 320x320 to 608x608 during training and a
the experiments are conducted in several constraints: specific resolution is chosen randomly for the input batch at
the beginning of every iteration. The network is trained on 2
1) Object detector: Our object detector is not tested on a
GeForce GTX TITAN X Graphics Cards, the batch size is 6
mobile device, but a PC. Hence, we only evaluate the for training and is 2 for testing. Training IoU threshold is 0.3.
accuracy of our object detector, not its inference speed, The total number of trained epochs is 120, the learning rate is
though we expect that our detector is able to obtain real-time ranged and dropped gradually from 10-4 at the first 2 epochs
inference speed on mobile devices due to its compactness. to 10-6 at the end of the training.
2) Environmental condition classifier: Our system is
The proposed object detection scheme is evaluated in
validated on BDD100K dataset [9], this dataset does not terms of Average Precision (AP) metric. We follow the
supply image data in different weather conditions, we PASCAL VOC convention by reporting AP at IoU = 0.5
simulate the variation of surrounding environment contexts (AP50). Our network with input size 608x608 produces
by using the change of light condition in different times of favorable detection outcomes of 43.6 AP corresponding to
day as described in Fig. 2. 7,978 detected objects out of 13,262 ground-truth objects with
IoU = 0.5, and 60.4 AP corresponding to 10,608 detected
B. Object Detector
objects with IoU = 0.3 (AP30). In order to compare the results
Our object detection network, Backbone17-Det, is trained in [15] in terms of the AP50 metric, two-stage object detection
on BDD100K [9]. BDD100K is a large-scale dataset with over model Faster R-CNN [3] is trained and tested on BDD100K
100K videos and its 2D bounding box annotations include 10 person data. The AP for our proposed scheme is 43.6 while
object categories: bus, traffic light, traffic sign, person, bike, Faster R-CNN shows 45.4. Note that the input image
resolutions are 1000x600 and 608x608 for Faster R-CNN and
TABLE II. OBJECT DETECTION PERFORMANCE COMPARISON our proposed scheme, respectively. Table II shows the
performance comparison. Our person detection results are
Model AP30 (%) AP50 (%) True Positives Labels shown in Fig. 5, red boxes present the labels and the detection
Our 60.4 - 10,608 (80.0%) 13,262
boxes are indicated in green.

Our - 43.6 7,978 (60.2%) 13,262 C. Environmental Condition Classifier

Faster R-CNN - 45.4±5.2 - 13,262
Due to the experimental constraints explained above, we
deploy a simple classifier that recognizes the light condition

161
Authorized licensed use limited to: National Taipei Univ. of Technology. Downloaded on May 06,2025 at 06:40:26 UTC from IEEE Xplore. Restrictions apply.
 Algorithm: Light Condition Classification

Input: RGB image

Output: light condition
initialize threshold values {T1, T2, T3} // 0<T1 <T2 <T3 <255
while system is running do
convert RGB input image to grayscale
compute average gray value (AGV)
if AGV ≥ T1 then
light_condition ← daytime
else if T2 ≤ AGV < T1 then
light_condition ← twilight
else if T3 ≤ AGV < T2 then
light_condition ← nighttime with street light
else
light_condition ← nighttime without street light
end if
return light_condition
end while
Fig. 7. Detection result is displayed on web browser.
Fig. 6. Pseudocode of light condition classification algorithm.
potential for the scalability of the next generation of
of the surrounding environment using an average gray value autonomous driving cars.
(AGV) of the grayscale of the RGB input image. We simulate Although the experimental results have shown the
our classifier with 4 light conditions as mentioned in Section contribution of the proposed system, there exist room for
II. We empirically choose 3 thresholds T1, T2, and T3 of the improvement. Important aspects for future works include (1)
AGVs to determine the light condition output, that is, if AGV model quantization and knowledge distillation for enhancing
≥ T1, the captured scene is in daytime, if T2 ≤ AGV < T1, the the compactness of the object detection model and improving
captured scene is in twilight, if T3 ≤ AGV < T2, the captured detection accuracy, and (2) validating the proposed system
scene is in nighttime with street light, otherwise, the captured with other datasets with different environmental contexts.
scene is in nighttime without street light. We empirically
choose T1 = 50, T2 = 20, and T3 = 10. The pseudocode of the REFERENCES
light condition classification algorithm is shown in Fig. 6. [1] J. Redmon, and A. Farhad, “YOLO9000: Better, Faster, Stronger,” in
2017 IEEE Conference on Computer Vision and Pattern Recognition
The detected bounding boxes should be drawn on the input (CVPR), Honolulu, HI, USA, 2017.
image for the sake of visualization and the result is displayed [2] Joseph Redmon and Ali Farhadi, “YOLOv3: An Incremental
on a built-in monitor via a web browser. To this end, we create Improvement”, in arXiv preprint arXiv:1804.02767, 2018.
a web application using Django web framework and establish [3] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, “Faster R-
a connection between the system and the cloud server for the CNN: Towards Real-Time Object Detection with Region Proposal
purposes of surveillance and dealing with critical situations as Networks,” in Neural Information Processing Systems (NIPS 2015),
Montreal, Quebec, Canada, 2015.
explained throughout the paper. Therefore, our proposed
system requires an acceptable internet connection. For the [4] Le-Anh Tran, Ngoc-Phu Le, Truong-Dong Do, and My-Ha Le, “A
Vision-based Method for Autonomous Landing on a Target with a
sake of efficiency, the system does not always send image data Quadcopter,” in 4th International Conference on Green Technology
to the server, instead, object detection results are merely and Sustainable Development (GTSD 2018), Ho Chi Minh City,
displayed on the built-in monitor and the system only sends a Vietnam, Nov. 23-24, 2018, pp. 601-606.
warning to the server whenever the classifier recognizes a [5] Le-Anh Tran, and My-Ha Le, “Robust U-Net-based Road Lane
significant change of the environmental condition, thus it does Markings Detection for Autonomous Driving,” in 2019 International
not require a high-quality internet connection, however, the Conference on System Science and Engineering (ICSSE 2019), Quang
Binh, Vietnam, Jul. 19-21, 2019, pp. 62-66.
better response speed, the safer the self-driving car. The web
[6] Le-Anh Tran, “Towards Improvement of Object Detection Accuracy
application with a simple interface is depicted in Fig. 7. for Deep Convolutional Neural Networks”, Master's Thesis, Myongji
University, 2021.
V. CONCLUSIONS AND FUTURE WORKS
[7] Truong-Dong Do, Minh-Thien Duong, Quoc-Vu Dang, and My-Ha Le,
In this work, we propose a novel enhancement for the “Real-Time Self-Driving Car Navigation Using Deep Neural
robustness of the object detection module in advanced driver Network,” in 4th International Conference on Green Technology and
Sustainable Development (GTSD 2018), Ho Chi Minh City, Vietnam,
assistance systems (ADAS). A unified system integrating an Nov. 23-24, 2018, pp. 7-12.
efficient object detector and an environmental condition [8] A. Ziebinski, R. Cupek, H. Erdogan, and S. Waechter, “A Survey of
classifier is proposed. As object detectors have become an ADAS Technologies for the Future Perspective of Sensor Fusion,” in
essential component in robotic systems, their performance 2016 International Conference on Computational Collective
during the deployment phase is critical to ensure the safety and Intelligence (ICCCI), Halkidiki, Greece, 2016, pp. 135–146.
reliability of the whole system. Since, however, modern self- [9] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan,
driving cars are still so far from being applied widely due to V., and Darrell, T., “BDD100K: A Diverse Driving Dataset for
Heterogeneous Multitask Learning,” in 2020 IEEE/CVF Conference
their limitation of accuracy, more robust and stable ADAS on Computer Vision and Pattern Recognition (CVPR), Seattle, WA,
should be developed. To this end, our system is proposed to USA, 2020.
improve the safety of a self-driving car itself and to enhance [10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld,
ADAS efficiency with reliable human supports from the Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and
server. The proposed system in this paper can be considered a Bernt Schiele, “The Cityscapes Dataset for Semantic Urban Scene

162
Authorized licensed use limited to: National Taipei Univ. of Technology. Downloaded on May 06,2025 at 06:40:26 UTC from IEEE Xplore. Restrictions apply.
 Understanding,” in 2016 Conference on Computer Vision and Pattern the European Conference on Computer Vision (ECCV), Munich,
Recognition (CVPR), United States, 2016, pp. 3213-3223. Germany, 2018, pp. 3-19.
[11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for [14] Q. M. Rahman, N. Sünderhauf, and F. Dayoub, “Online Monitoring of
Image Recognition,” in 2016 IEEE Conference on Computer Vision Object Detection Performance Post-Deployment”, in arXiv preprint
and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016. arXiv:2011.07750, 2020,
[12] J. Hu, L. Shen, and G. Sun, “Squeeze-and-Excitation Networks,” in [15] Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern, “Predictive
2018 IEEE/CVF Conference on Computer Vision and Pattern Inequity in Object Detection”, in arXiv preprint arXiv:1902.11097,
Recognition (CVPR), Salt Lake City, UT, USA, 2018. 2019.
[13] Sanghyun Woo , Jongchan Park , Joon-Young Lee, and In So Kweon,
“CBAM: Convolutional Block Attention Module,” in Proceedings of

163
Authorized licensed use limited to: National Taipei Univ. of Technology. Downloaded on May 06,2025 at 06:40:26 UTC from IEEE Xplore. Restrictions apply.