Final Project Report
Final Project Report
on
Submitted by
Project Members
1
SCHOOL OF COMPUTER ENGINEERING AND TECHNOLOGY
C E R T I FI CAT E
This is to certify that, Aneesh Pol, Muskaan Rajput, Sakshi Reddy, Swarali Paygude
of BTech.( Computer Science & Engineering) have completed their project titled
“Automated PCB Fault Detection System[Comparative Study of Image Processing and
Transfer Learning Technique]” and have submitted this Capstone Project Report
towards fulfillment of the requirement for the Degree-Bachelor of Computer
Science & Engineering (BTech-CSE) for the academic year 2020-2021.
Internal Examiner:
External Examiner:
Date:
2
Acknowledgement
We would like to take this opportunity to thank our internal guide Prof. Madhura Phatak ma’am
for giving us all the help and guidance we needed. We are extremely grateful to her for her kind
support and patience. Her valuable suggestions helped in shaping this project and giving us the
direction to move ahead.
We are also grateful to Prof. Dr. Vrushali Kulkarni, Head, School of Computer Engineering &
Technology for this opportunity and her encouragement.
Swarali Paygude,
Muskaan Rajput,
Sakshi Reddy,
Aneesh Pol
3
Abstract
Printed circuit boards (PCBs) are a crucial component in most electronic devices. Accurate
PCB manufacturing is critical as manufacturers are required to produce PCBs in large
quantities. Therefore, maintaining the quality of such large numbers of PCBs is challenging.
Automated inspection systems can prove helpful in quality maintenance. Such systems
overcome the limitations of manual inspection for a large number of PCBs. Automated visual
PCB inspection can provide fast detection of defects and therefore can prove to be an asset in
the manufacturing process. This project aims to achieve fault detection of bare PCBs through
two different methods; one being the traditional algorithmic approach using image processing,
which involves the use of image subtraction method and the other one being a transfer learning
approach, which involves the pre-trained VGG16 model. A comparative study of both the
methods is done.
4
List of Figures
1 System Block Diagram…………………………………...……… 15
2 Gantt Chart ……………………………………………………...….. 17
3 Project Plan…………………………………………………...….. 17
4 Augmented Images………………………………………….……….19
5 Image Processing Module……………………………………..……...20
6 Transfer Learning Module…………………………………..……….21
7 After Image Subtraction(Missing hole).............................................24
8 After Median Filtering(Missing hole)................................................24
List of Tables
1 Literature Survey ……………………………………………………..11
2 Original Dataset of Bare PCB images……………………………....18
3 Result Table for defect detection(Image Subtraction)........................24
4 Result Table for defect detection(VGG-16)......................................25
Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .…….……. ..I
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .….…… …II
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ….. . ….…... . III
5
1 Introduction 7
1.2 Area 7
2 Literature Survey 9
3 Problem Statement 11
4 Project Requirements 13
4.1 Resources 13
6
6 Project Plan 16
7 Implementation 17
7.2 Methodology 18
7.3 Algorithm 20
8 Performance Evaluation 22
Applications 26
Conclusion 27
References 29
Publication Details 30
7
Chapter 1
Introduction
1.1 Project statement
To develop an image based fault detection system to identify various defects on bare Printed
Circuit Boards (PCBs)
[a comparative study of algorithmic computations of image processing and transfer learning approaches]
1.2 Area
PCB fault detection can be achieved using a variety of techniques. In our project we aim
to use two techniques for pcb fault detection and classification. The first method is Transfer
Learning. Transfer learning falls under the domain of machine learning. In transfer learning a
model is developed for a particular task and is then reused as the starting point for another task.
The pre-trained model which we would be using is VGG16.
The second method which we would be implementing in our project is a traditional
handcrafted unsupervised algorithm. Image processing is an extremely large domain and
consists of a number of methods and techniques within it. Image processing is basically a
method adopted to perform certain operations on the image in order to extract useful
information from it.
8
components are mounted on the bare PCB. This gives rise to a number of soldering defects. In
our project we are focusing on detection of faults occurring on bare pcbs. In the past a number
of supervised as well as unsupervised methods were adopted for detection of faults on bare
PCB.
In this project we will be using two methods viz transfer learning which is a supervised
method and image processing techniques which is an unsupervised method and then compare
the two on the basis of accuracy and time. Accurate PCB manufacturing is critical as
manufacturers are required to produce PCBs in large quantities. Maintaining the quality of such
large numbers of PCBs is challenging. Automated inspection systems can prove helpful in
quality maintenance. Such systems overcome the limitations of manual inspection for a large
number of PCBs. Automated visual PCB inspection can provide fast detection of defects and
therefore can prove to be an asset in the manufacturing process.
9
Chapter 2
Literature Survey
In the development of technology, image processing and computer vision have experienced rapid
development in recent years. Rather than the manual inspection computer vision technique is playing an
important role in the advanced automatic industry. In order to reduce the cost in manufacturing of electronic
boards, defects in the bare PCB must be identified. Many researchers have done a lot of work regarding PCB
inspection.
Numerous methods are suggested by the researchers on PCB defect classification. In this section, brief
evaluations of a few essential contributions to the existing works of literature are presented.
1. Study of the Image International 3 approaches: Template Some of the defects cannot be Pratiksha R.
Processing Journal of matching, Image addressed individually. Masalkar,
algorithms for Engineering subtraction, Image During grayscale to binary Prabha S.
defect detection of Technology morphology to detect transformation, undesirable Kasliwal
PCBs Science and defects and classify them images may be produced by
Research into groups. the noise.
(IJETSR) ,2017 Dataset was created using Future scope includes
Pi camera for capturing inspection and analysis of a
images. PCB with Surface Mounted
Devices.
10
3 Automatic PCB International Detect and classify all the Further study on robust Prachi P.
Defects Detection Journal of known 14 defects technique to differentiate the Londe, S.
and Classification Current Used MATLAB defects on single PCB image. A. Chavan
using Matlab Engineering and Image subtraction method Study further defect detection
Technology for defect detection and classification on loaded
2014 KNN classification PCB
algorithm for
classification of defects
11
Chapter 3
Problem Statement
Our project aims to prepare a comparative study between transfer learning, with the help of a
pre-trained object detection model (VGG-16) and traditional handcrafted algorithm using
Image Processing Algorithms such as Image subtraction, to detect defects in bare PCBs.
The dataset contains labelled images of 12 different Printed Circuit Boards having six types of
defects (Missing hole, Mouse bite, Open circuit, Short, Spur, Spurious copper). Using these
images the model will be trained to detect these defects in real-time.
While the project uses a pre-trained model, it has to be modified to fit our purpose. The output
layers are changed to detect the faults and classify them accordingly.
The project aims to locate and classify multiple defects on a single PCB simultaneously and
make bounding boxes to visualize them.
Image processing algorithms such as Image subtraction are considered to be generally less
effective than CNN based algorithms but considering the rather small size of the dataset they
might return a better accuracy. Hence the comparative study might prove effective to
understand the importance of choosing the right algorithm, with regard to the size of the
dataset, in the accuracy of the model.
Although CNN based models are widely used for object detection in images, due to the small
size of the available dataset, the accuracy of the model could be affected adversely.
The lack of a standard dataset makes the comparison of our model with the accuracy of other
models difficult.
The transfer learning model (VGG16) was originally designed to detect a fixed number of
classes (objects) in images which did not include Printed circuit boards, hence the model has to
be modified accordingly to achieve best results.
12
3.3 Project Objectives
13
Chapter 4
Project Requirements
4.1 Resources
Software:
● VGG16 model
● Python (Libraries like OpenCV, Keras )
● Jupyter Notebook
● GIMP (image editor)
Hardware:
Laptop/PC
Risks Solutions
Low quality PCB images Train model with different kinds of images
consisting of noise. Take images with proper
illumination.
Failure to detect the defect. Train your model with a bigger and proper dataset.
Failure to procure a dataset with Create your own dataset. Image augmentation,
reasonable amounts of images. rotation, duplicating.
14
Chapter 5
System Analysis Proposed Architecture
15
● The images used for fault detection in pcb need to be clear inorder to achieve better
results.
● The developed system is specific to our dataset, hence will not provide the desired
accuracy on a different dataset.
● The dataset consists of more defective images as compared to non defective images.
16
Chapter 6
Project Plan
17
Chapter 7
Implementation
Image Processing
Missing Hole 20 10 10 5 10 55
Mouse Bite 20 10 10 5 10 55
Open Circuit 20 10 10 6 10 56
60 30 30 16 30 166
We also performed rotations of all images at 90, 180 and 270 degrees from the original orientation. As a
result, we had 4 times the total number of original images, ie, 166 * 4 = 664.
● Image Augmentation - Image Augmentation is used to artificially expand the existing dataset by developing
a variety of versions of each image present. Image augmentation is performed to increase the number of images which
18
inturn helps train the model more efficiently. Image can be processed in a number of ways such as, horizontal flip,
vertical flip, zoom in, zoom out, contrast, rotation at specific or random angles etc. This is achieved using Image Data
Generator API from Keras. Hence using this technique we created 20 random augmented images for each of the
images present in the original dataset.
7.2 Methodology
Image Subtraction
Image processing is a method to perform certain operations on an image, which will result in an enhanced image, or
extracting some useful information from it. In this type of processing, the input is an image and the output may be an
image or characteristics/features associated with that image. Image processing basically includes the following three
steps:
● Importing the image via image acquisition tools
● Analysing and manipulating the image
● Output to show the results
The approach used in this system is based on the Image Subtraction technique, in order to detect the PCB defects from
the images of bare PCBs. Image subtraction is one of the image processing techniques, also known as pixel subtraction.
In this technique, the digital numeric values of pixels from an image are subtracted from the values of pixels of another
image. This is done for the purpose of detecting the changes between the two images. The functions used within the
system are taken from the OpenCV library of Python. Initially, pre-processing is done on the template and test images,
to resize them, so that both are of the same size. This is necessary, since pixel-to-pixel mapping is done during image
subtraction. Both the images should also be of the same orientation. Then, they are converted to grayscale images. After
that, both images are blurred using a blurring method, and adaptive thresholding is performed on them. Finally, an
image subtraction function is applied, which subtracts the test image from the template image, to get the result as the
difference image. This image contains the defective part, which is shown as bright(white) pixels on a dark(black)
background. We can thus observe the defective part, and easily see where the defects are located. Contour detection is
19
also applied to count the number of defects detected in the test image. This can further be used while calculating
accuracy of the system.
Transfer Learning
Transfer learning allows solutions to multiple problems and applications. It is primarily a technique in which any neural
network model is initially trained on a similar dataset to that of the actual problem. On completion of stage one, certain
layers from the then trained model are retrained to generate a new model addressing the required problem statement. A
model is initially trained using a larger dataset. The trained layers are frozen and used as it is while addressing a
problem statement with a comparatively smaller dataset. In such cases only certain layers, usually the fully connected
and output layers are modified as per the problem statement. Hence using this technique models can be trained using
smaller datasets as well without compromising the accuracy. Some popular models include ResNet-50, VGG-16,
VGG-19, Inception V3, XCeption. Various word embedding models include Word2Vec, GloVe, FastText. The
pre-trained model which we have used for pcb fault detection is VGG-16. Using pre-trained models is recommended
when one has a large dataset in hand. The original VGG-16 architecture has 2 fully connected layers followed by a
softmax for output. The 16 in VGG-16 refers to it having 16 layers that have weights. The network is a pretty large
network and it has about 138 million parameters. All the hidden layers use ReLU as its activation function. ReLU is
more computationally efficient because it results in faster learning and it also decreases the likelihood of vanishing
gradient problem. When using VGG-16 there is no requirement of a reference image. For training we used Transfer
Learning in Keras on VGG-16. First and foremost we imported keras and all the methods and functions required to
build and train the model. Inorder to import all the images we used ImageDataGenerator from keras. We had to resize
the images to 224,224,3 as the model was originally trained using this size. We then import VGG-16 from keras with
pre-trained weights which were originally trained on imagenet.
Once we have downloaded the model, the model now needs to be modified as per our problem statement. This needs to
be done because pcb detection does not belong to any of the 1000 classes which was used originally. Next, there were in
all 1000 classes, as per our problem statement we have only 3 classes which are missing holes, open circuit and mouse
bite. The activation function used was ReLu in the original model. In case of more classes we can also use the softmax
activation function. The output layer consists of 1000 neurons, we will have to modify it to 3 output neurons. Along
20
with this we add a flatten layer as well.Whatever output we receive we condense that into one dimension using the
flatten layer. The model was trained twice. Once using the original dataset and then using the dataset consisting of the
augmented images. Different combinations of epochs were experimented with to attain maximum accuracy.
7.3 Algorithm
Image Processing
21
ResultantImg = cv2.subtract(img1,img2)
● Detection - The three types of defects detected are: MissingHole, MouseBite and OpenCircuit. The subtracted image
is passed through a median filter to remove the noise. Following is the syntax of this method:
medianBlur(src, dst, ksize)
The image thus obtained can detect the different defects.
● Contour Detection - In OpenCV, finding contours is like finding white object from black background.
The contours are a useful tool for shape analysis and object detection and recognition. We used this to detect the
number of defects in an image. The method used for this purpose is:
cv2.findContours()
Transfer Learning
VGG16 Methodology
1. Get path of the directory containing dataset
2. Using os, numpy and image libraries convert the image dataset into a numpy array of images resized to
(224,224)
3. Roll axis and transform array into the shape (no. of images, 224,244,3)
4. Label the data and use one hot encoding
5. Shuffle and Split the data into training and testing
6. Use Pretrained VGG16 model from keras without including top
7. Using the output from the convoluted block 5 make custom fully connected layers and output layer of the
number of required classes
8. Set the layers till convolution block 4 as not trainable and the rest as trainable
9. Compile model using adam optimizer and categorical cross entropy
10. Fit model to the train test split dataset
22
Chapter 8
Performance Evaluation
For evaluating the performance of image Accuracy: This metric creates two local
subtraction technique, we used a simple variables, total and count that are used to
percentage calculation method to find the compute the frequency with which y_pred
accuracy of detection of the defects. matches y_true. This frequency is ultimately
The formula used is as follows: returned as binary accuracy: an idempotent
acc_percent = operation that simply divides total by count.
(total_detected/total_defects)*100 Loss: Categorical cross entropy, it is a Softmax
where total_detected is the total number of activation plus a Cross-Entropy loss. If we use
defects detected in an image, and total_defects this loss, we will train a CNN to output a
is the total number of defects actually present probability over the C classes for each image.
in that image. It is used for multi-class classification.
23
Chapter 9
Result and Analysis
● Image Subtraction
We implemented the handcrafted algorithm of image subtraction on all the images in our
dataset where we subtracted the test image from the reference image of five different PCBs.
We then analysed the result by finding the accuracy of the detection using a simple percentage
calculation method. The average detection accuracies of missing hole, mouse bite and open
circuit were found to be 86.05, 72.64 and 80.43 respectively.
Fig7. After Image Subtraction (Missing hole) Fig8. After Median Filtering (Missing hole)
24
● Transfer Learning- VGG-16
VGG16 Transfer Learning model was implemented on the augmented dataset to achieve
these results. The results returned by the model are illustrated in a tabular format depicting
the number of Epochs that the model was trained for, the number of steps completed in each
epoch, the loss calculated by binary cross entropy and finally the accuracy that was obtained.
25
Applications
● This project can be used to detect defects such as missing holes, mouse bite and open
circuit on bare PCBs.
● Educational institutes can use the system to maintain the PCBs used in their
laboratories
● PCB manufacturing industry can use this system for Quality Assurance
● Electronic gadgets which extensively use PCBs can also benefit from this system
26
Conclusion
In this project, we have tried to develop a system that would be helpful in the field of PCB
manufacturing. Our prototype models will detect and classify bare PCB defects, which will
help in minimizing the waste of defective PCBs which are otherwise discarded altogether.
The comparative study will help in identifying the better and more accurate approach among
the two implemented, according to the circumstances and available resources. The dataset
used is an open-source dataset, which can later be benchmarked and people can use this
prototype along with the dataset, for further research and development.
27
Future prospects of the project
In the future work, such a dataset can be used which includes different types(designs) of
PCBs, i.e. a variety of PCB images. Also, a dataset with a large number of images can be
used. For an unbiased result, a more balanced dataset can be used, which contains equal
amounts of defected and non-defected images.
The system can be modified to incorporate detection of more PCB defects. The system can
also be improvised in such a way that real-time defect detection is possible.
An integrated system can be developed in which the defects present on the bare PCB can be
detected and localized using image subtraction and the defects can further be classified using
a pre-trained model.
One more prospect can be to experiment using different pre-trained models, and other
handcrafted algorithms.
28
References
[1] Pratiksha R. Masalkar, Prabha S. Kasliwal, “Study of the Image Processing algorithms for defect
detection of PCBs”, International Journal of Engineering Technology Science and Research, June
2017
[2] Harshitha R, Apoorva G C, Ashwini M C, Kusuma T S, “Components Free Electronic Board
Defect Detection and Classification Using Image Processing Technique”, International Journal of
Engineering Research & Technology ,2018
[3] Daniel Katz Bonello, Yuzo Iano, Umberto Bonello Neto, “A New Based Image Subtraction
Algorithm for Bare PCB Defect Detection”, International Journal Multimedia and Image
Processing, September 2018
[4] G. Prathima, A. Yasaswini Naga Lakshmi, et. al., “Defect Detection in PCB Using Image
Processing”, International Journal of Advanced Science and Technology, 2020
[5] Ms. Vinita P, Mr. shivkant kaushik, “PCB Fault Detection by Image Subtraction Method”,
International Journal of Scientific Engineering and Applied Science (IJSEAS), January 2016
[6] Karen Simonyan & Andrew Zisserman, “Very deep convolutional networks for large-scale image
recognition”, ICLR, 2015
[7] Fa'iq Raihan, Win Ce, “PCB Defect Detection USING OPENCV with Image Subtraction
Method”, IEEE, 2017
[8] Ihar Volkau, Abdul Mujeeb, Dai Wenting, Erdt Marius, Sourin Alexei, “Detection Defect in
Printed Circuit Boards using Unsupervised Feature Extraction Upon Transfer Learning”, IEEE,
2019
[9] Leandro H. de S. Silva, George O. de A. Azevedo, Bruno J. T. Fernandes, Byron L D Benzerra,
“Automatic Optical Inspection for Defective PCB Detection Using Transfer Learning.”, IEEE,
2019
[10] Ajay Pal Singh Chauhan, Sharat Chandra Bhardwaj, “Detection of Bare PCB Defects by Image
Subtraction Method using Machine Vision”, Proceedings of the World Congress on Engineering
2011 Vol II
[11] Jithendra P R Nayak, Parameshachari B D, et. al., “Identification of PCB Faults using Image
Processing”, IEEE, 2017
[12] Prachi P. Londe, S. A. Chavan, “Automatic PCB Defects Detection and Classification using
Matlab”, International Journal of Current Engineering and Technology 2014
29
Publication Details
Our results-based paper(manuscript) on our project was submitted to and accepted by the
following international journals:
➢ The International Organization of Scientific Research (IOSR) Journal
➢ International Journal for Research in Applied Science and Engineering Technology
(IJRASET)
Published the paper titled “Comparative Study of Image Processing and Transfer Learning
Techniques for an Automated PCB Fault Detection System” at IJRASET Volume 9, Issue VI,
June 2021.
30
Appendices
A. Base Paper(s)
1. Study of the Image Processing algorithms for defect detection of PCBs
2. Very deep convolutional networks for large-scale image recognition
B. Plagiarism Report from any open source/proprietary source
CHAPTER 1
CHAPTER 3
31
CHAPTER 4
CHAPTER 5
CHAPTER 7
32
CHAPTER 8
CHAPTER 9
33