Machine Vision and Applications (2021) 32:84

https://doi.org/10.1007/s00138-021-01191-9

ORIGINAL PAPER

Segmentation of photovoltaic module cells in uncalibrated

electroluminescence images
Sergiu Deitsch1 · Claudia Buerhop-Lutz2 · Evgenii Sovetkin3 · Ansgar Steland4 · Andreas Maier5 ·
Florian Gallwitz6 · Christian Riess7

Received: 6 August 2019 / Revised: 3 December 2020 / Accepted: 5 March 2021

© The Author(s) 2021

Abstract
High resolution electroluminescence (EL) images captured in the infrared spectrum allow to visually and non-destructively
inspect the quality of photovoltaic (PV) modules. Currently, however, such a visual inspection requires trained experts to
discern different kinds of defects, which is time-consuming and expensive. Automated segmentation of cells is therefore a
key step in automating the visual inspection workflow. In this work, we propose a robust automated segmentation method
for extraction of individual solar cells from EL images of PV modules. This enables controlled studies on large amounts of
data to understanding the effects of module degradation over time—a process not yet fully understood. The proposed method
infers in several steps a high-level solar module representation from low-level ridge edge features. An important step in the
algorithm is to formulate the segmentation problem in terms of lens calibration by exploiting the plumbline constraint. We
evaluate our method on a dataset of various solar modules types containing a total of 408 solar cells with various defects. Our
method robustly solves this task with a median weighted Jaccard index of 94.47% and an F1 score of 97.62%, both indicating
a high sensitivity and a high similarity between automatically segmented and ground truth solar cell masks.

Keywords PV modules · EL imaging · Visual inspection · Lens distortion · Solar cell extraction · Pixelwise classification

1 Introduction
B Sergiu Deitsch
[email protected] Visual inspection of solar modules using EL imaging allows
Claudia Buerhop-Lutz to easily identify damage inflicted to solar panels either by
[email protected] environmental influences such as hail, during the assem-
Evgenii Sovetkin bly process, or due to prior material defects or material
[email protected] aging [5,10,65,90,91,93]. The resulting defects can notably
Ansgar Steland decrease the photoelectric conversion efficiency of the mod-
[email protected] ules and thus their energy yield. This can be avoided by
Andreas Maier continuous inspection of solar modules and maintenance
[email protected] of defective units. For an introduction and review of non-
Florian Gallwitz
[email protected]
Christian Riess 4 Institute of Statistics, RWTH Aachen University,
[email protected] Wüllnerstr. 3, 52062 Aachen, Germany
1 5 Pattern Recognition Lab, Friedrich-Alexander University
Pattern Recognition Lab, Friedrich-Alexander University
Erlangen-Nürnberg, Martensstr. 3, 91058 Erlangen, Germany Erlangen-Nürnberg, Martensstr. 3, 91058 Erlangen, Germany
2 6 Faculty of Computer Science, Nuremberg Institute of
Helmholtz-Institut Erlangen-Nürnberg HI ERN,
Forschungszentrum Jülich GmbH, Immerwahrstr. 2, 91058 Technology, Keßlerplatz 12, 90489 Nürnberg, Germany
Erlangen, Germany 7 IT Security Infrastructures Lab, Friedrich-Alexander
3 IEK5-Photovoltaik, Forschungszentrum Jülich GmbH, 52425 University Erlangen-Nürnberg, Martensstr. 3, 91058
Jülich, Germany Erlangen, Germany

0123456789().: V,-vol 123

 84 Page 2 of 23 S. Deitsch et al.

automatic processing tools for EL images, we refer to Mauk spective distortions. This distinguishes the manufacturing
[59]. setting from acquisitions in the field, where PV modules
An important step towards an automated visual inspection may be occluded by cables and parts of the rack, and the
is the segmentation of individual cells from the solar module. perspective may be strong enough to require careful cor-
An accurate segmentation allows to extract spatially normal- rection. However, perspective distortion also makes it more
ized solar cell images. We already used the proposed method difficult to identify defective areas (e.g., microcracks) due
to develop a public dataset of solar cells images [12], which to the foreshortening effect [4]. Therefore, capturing EL
are highly accurate training data for classifiers to predict images from an extreme perspective is generally not advis-
defects in solar modules [18,60]. In particular, the Convolu- able. Specifically for manufacturing environments, however,
tional Neural Network (CNN) training is greatly simplified the proposed method yields a robust, highly accurate, and
when using spatially normalized samples, because CNNs are completely automatic segmentation of solar modules into
generally able to learn representations that are only equiv- solar cells from high resolution EL images of PV modules.
ariant to small translations [35, pp. 335–336]. The learned Independently of the setting, our goal is to allow for some
representations, however, are not naturally invariant to other flexibility for the user to freely position the camera or use
spatial deformations such as rotation and scaling [35,44,52]. zoom lenses without the need to recalibrate the camera.
The identification of solar cells is additionally required With this goal in mind, a particular characteristic of the
by the international technical specification IEC TS 60904- proposed segmentation pipeline is that it does not require an
13 [42, Annex D] for further identification of defects on cell external calibration pattern. During the detection of the grid
level. Automated segmentation can also ease the develop- that identifies individual solar cells, the busbars and the inter
ment of models that predict the performance of a PV module solar cell borders are directly used to estimate lens distortion.
based on detected or identified failure modes, or by deter- Avoiding the use of a separate calibration pattern also avoids
mining the operating voltage of each cell [70]. The data the risk of an operator error during the calibration, e.g., due
describing the cell characteristics can be fed into an elec- to inexperienced personnel.
tric equivalent model that allows to estimate or simulate the A robust and fully automatic PV module segmentation
current-voltage characteristic (I-V) curve [13,46,72] or even can help understanding the influence of module degradation
the overall power output [47]. on module efficiency and power generation. Specifically, this
The appearance of PV modules in EL images depends on a allows to continuously and automatically monitor the degra-
number of different factors, which makes an automated seg- dation process, for instance, by observing the differences in
mentation challenging. The appearance varies with the type a series of solar cell images captured over a certain period of
of semiconducting material and with the shape of individual time. The segmentation also allows to automatically create
solar cell wafers. Also, cell cracks and other defects can intro- training data for learning-based algorithms for defect classi-
duce distracting streaks. A solar cell completely disconnected fication and failure prediction.
from the electrical circuit will also appear much darker than a
functional cell. Additionally, solar modules vary in the num-
ber of solar cells and their layout, and solar cells themselves 1.1 Contributions
are oftentimes subdivided by busbars into multiple segments
of different sizes. Therefore, it is desirable for a fully auto- To the best of our knowledge, the proposed segmentation
mated segmentation to infer both the arrangement of solar pipeline is the first work to enable a fully automatic extraction
cells within the PV module and their subdivision from EL of solar cells from uncalibrated EL images of solar modules
images alone, in a way that is robust to various disturbances. (cf., Fig. 1b). Within the pipeline, we seek to obtain the exact
In particular, this may ease the inspection of heterogeneous segmentation mask of each solar cell through estimation of
batches of PV modules. nonlinear and linear transformations that warp the EL image
In this work, we assume that EL images are captured in into a canonical view. To this end, our contributions are three-
a manufacturing setting or under comparable conditions in a fold:
test laboratory where field-aged modules are analyzed either
regularly or after hazards like hailstorms. Such laboratories
oftentimes require agile work processes where the equip- 1. Joint camera lens distortion estimation and PV module
ment is frequently remounted. In these scenarios, the EL grid detection for precise solar cell region identification.
irradiation of the solar module predominates the background 2. A robust initialization scheme for the employed lens dis-
irradiation, and the solar modules are captured facing the EL tortion model.
camera without major perspective distortion. Thus, the geo- 3. A highly accurate pixelwise classification into active solar
metric distortions that are corrected by the proposed method cell area on monocrystalline and polycrystalline PV mod-
are radial lens distortion, in-plane rotation, and minor per- ules robust to various typical defects in solar modules.

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 3 of 23 84

Row 1

Row 2
...
... ...

(a) (b)
Fig. 1 (a) An EL image of a PV module overlaid by a rectangular grid ( ) and parabolic curve grid ( ) including the busbars ( )
determined using our approach. The intersections of the rectangular grid were registered to curve grid intersections to accurately align both grids.
Notice how the rectangular grid is still not able to capture the curved surface of the solar module induced by the (weak) lens distortion that increases
especially towards the image border. Using the curve grid, we estimate the lens distortion, rectify the image and finally extract the individual cells
using the estimated module topology (b). The segmented solar cells can be used for further analysis, such as automatic defect classification or
failure prediction in PV modules. The solar cells are approximately 15.6 cm × 15.6 cm with a standard 60 cell PV module with overall dimensions
of 1 m × 1.65 m

Moreover, our method operates on arbitrary (unseen) module constraint exploits the fact that the projection of straight
layouts without prior knowledge on the layout. lines under radial and tangential distortion will not be truly
straight. For example, under radial distortion, straight lines
1.2 Outline are images as curves. For typical visual inspection tasks,
a single image is sufficient to estimate the lens distor-
The remainder of this work is organized as follows. Section tion parameters [2,16,17,20,25,78]. This can be achieved by
2 discusses the related work. In Sect. 3, the individual stages decoupling the intrinsic parameters of the camera from the
of the segmentation pipeline are presented. In Sect. 4, we parameters of the lens distortion model [20].
evaluate the presented segmentation approach on a number Novel methodologies employ CNNs for various segmen-
of different PV modules with respect to the segmentation tation tasks. Existing CNN-based segmentation tasks can
accuracy. Finally, the conclusions are given in Sect. 5. be categorized into (1) object detection, (2) semantic seg-
mentation, and (3) instance-aware segmentation. One of the
first CNN object detection architectures is Regions with
2 Related work CNN features (R-CNN) [32] to learn features that are subse-
quently classified using a class-specific linear Support Vector
The segmentation of PV modules into individual solar cells Machine (SVM) to generate region proposals. R-CNN learns
is related to the detection of calibration patterns, such as to simultaneously classify object proposals and refine their
checkerboard patterns commonly used for calibrating intrin- spatial locations. The predicted regions, however, provide
sic camera and lens parameters [29,36,41,69,79]. However, only a coarse estimation of object’s location in terms of
the appearance of calibration patterns is typically perfectly bounding boxes. Girshick [31] proposed Fast Region-based
known, whereas detection of solar cells is encumbered by Convolutional Neural Network (Fast R-CNN) by acceler-
various defects that are a priori unknown. Additionally, the ating training and testing times while also increasing the
number of solar cells in a PV module and their layout can detection accuracy. Ren et al. [75] introduced Region Pro-
vary. We also note that existing lens models generally assume posal Network (RPN) that shares full-image convolutional
wide angle lenses. However, their application to standard features with the detection network enabling nearly cost-
lenses is to our knowledge not widely studied. free region proposals. RPN is combined with Fast R-CNN
To estimate the parameters of a lens distortion model, into a single network that simultaneously predicts object
the plumbline constraint is typically employed [11]. The bounds and estimates the probability of an object for each

123
 84 Page 4 of 23 S. Deitsch et al.

proposal. For semantic segmentation, Long et al. [56] intro- estimate lens distortion parameters, we constrain the prob-
duced Fully Convolutional Networks (FCNs) allowing for lem using domain knowledge by operating on a coherent
pixelwise inference. The FCN is learned end-to-end and grid. This joint methodology allows to correct errors through
pixels-to-pixels requiring appropriately labeled training data. feedback from the optimization loop used for estimating lens
Particularly, in medical imaging the U-Net network architec- model parameters. The proposed approach conceptually dif-
ture by Ronneberger et al. [77] has been successfully applied fers from Sovetkin and Steland [86], where both steps are
for various segmentation tasks. In instance segmentation, Li decoupled and an external calibration is required.
et al. [51] combined segment proposal and object detection
for Fully Convolutional Instance Segmentation (FCIS) where
the general idea is to predict the locations in a fully convolu- 3 Methodology
tional network. He et al. [39] proposed a Mask R-CNN which
extends Faster R-CNN. The proposed framework uses a bottom-up pipeline to gradu-
The work by Mehta et al. [62] introduces a CNN for the ally infer a high-level representation of a solar module and its
prediction of power loss. Their system additionally localizes cells from low-level ridge edge features in an EL image. Cell
and classifies the type of soiling. Their work is based on RGB boundaries and busbars are represented as parabolic curves
images of whole PV modules and addresses the additional to robustly handle radial lens distortion which causes straight
geometric challenges of acquisitions in the field. In contrast, lines to appear curved in the image. Once we estimated
this work operates on EL images of individual cells of a PV the lens distortion parameters, the parabolas are rectified to
module, and in particular focuses on their precise segmenta- obtain a planar cell grid. This rectified representation is used
tion in a manufacturing setting. to segment the solar cells.
The main limitation of learning-based approaches is
the requirement of a considerable number of appropriately 3.1 Overview
labeled images for training. However, pixelwise labeling is
time-consuming, and in absence of data not possible at all. The general framework for segmenting the solar cells in EL
Also, such learning-based approaches require training data images of PV modules is illustrated in Fig. 2 and consists of
that is statistically representative for the test data, which the following steps. First, we locate the busbars and the inter
oftentimes requires to re-train a model on data with different solar cell borders by extracting the ridge edges. The ridge
properties. In contrast, the proposed approach can be read- edges are extracted at subpixel accuracy and approximated
ily deployed to robustly segment EL images of PV modules by a set of smooth curves defined as second-degree polyno-
without notable requirements of labeled training data. mials. The parametric representation is used to construct an
The closest work related to the proposed method was pre- initial grid of perpendicularly arranged curves that identify
sented by Sovetkin and Steland [86]. This method proposes a the PV module. Using this curve grid, we estimate the initial
robust PV module grid alignment for the application on field lens distortion parameters and hypothesize the optimal set of
EL images, where radial and perspective distortion, motion curves by further excluding outliers in a RANdom SAmple
blur, and disturbing background may be present. The method Consensus (RANSAC) scheme. Then we refine the lens dis-
uses an external checkerboard calibration for radial distortion tortion parameters that we eventually use to rectify the EL
correction, and prior knowledge on the solar cell topology in image. From the final set of curves we infer the PV module
terms of the relative distances of the grid lines separating configuration and finally extract the size, perspective, and
the busbars and cell segments. In contrast, EL images taken orientation of solar cells.
under manufacturing conditions may be cropped or rotated,
and the camera is not always pre-calibrated. Hence, the pro- 3.2 Preprocessing
posed method performs an automated on-line calibration for
every EL image. This is particularly useful for EL images First, the contrast of an EL image is enhanced to account for
of PV modules from various sources, for which the camera possible underexposure. Then, low-level edge processing is
parameters may not be available, or when zoom lenses are applied to attenuate structural variations that might stem from
used. Additionally, the proposed method performs a pixel- cracks or silicon wafer texture, with the goal of preserving
wise classification of pixels belonging to the active cell area larger lines and curves.
and therefore is able to provide masks tailored to a specific
module type. Such masks allow to exclude unwanted back- 3.2.1 Contrast enhancement
ground information and to simplify further processing.
In this work, we unify lens distortion estimation and grid Here, we follow the approach by Franken et al. [28]. A
detection by building upon ideas of Devernay and Faugeras copy Ibg of the input EL image I is blurred with a Gaus-
[20]. However, instead of using independent line segments to sian kernel, and a morphological closing with a disk-shaped

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 5 of 23 84

Preprocessing Curve Extraction Model Estimation Cell Extraction

Local ridge Quadratic Coherent Solar cell

features curves grid ROIs

(a) (b) (c) (d)

Fig. 2 The proposed PV module segmentation pipeline consists of four stages. In the preprocessing stage (a), local ridge features are extracted.
In the curve extraction stage (b), candidate parabolic curves are determined from ridges. In the model estimation stage (c), a coherent grid and the
lens distortion are jointly estimated. In the cell extraction stage (d) the cell topology is determined and the cells are extracted

structure element is applied. Dividing each pixel of I by Ibg the angle β(u) provide initial cues for ridge edges in the EL
attenuates unwanted background noise while emphasizing image (see Fig. 5c).
high contrast regions. Then, histogram equalization [34,
pp. 134 sqq.] is applied to increase its overall contrast. Fig-
ure 5b shows the resulting image I .
3.2.3 Contextual enhancement via tensor voting

3.2.2 Gaussian scale-space ridgeness Ridgeness can be very noisy (cf., Fig. 5c). To discern noise
and high curvatures from actual line and curve features,
The high-level grid structure of a PV module is defined by R(u) is contextually enhanced using tensor voting [61].
inter-cell borders and busbars, which correspond to ridges in Tensor voting uses a stick tensor voting field to model the
the image. Ridge edges can be determined from second-order likelihood that a feature in the neighborhood belongs to the
partial derivatives summarized by a Hessian. To robustly same curve as the feature in the origin of the voting field [27].
extract line and curve ridges, we compute the second-order The parameter ς > 0 controls the proximity of the voting
derivative of the image at multiple scales [54,55]. The field, and ν determines the angular specificity that we set to
responses are computed in a Gaussian pyramid constructed ν = 2 in our experiments.
from an input EL image [53]. This results in several layers of Following Franken et al. [27], stickness R̃(u) = λ̃1 − λ̃2
the pyramid at varying resolutions commonly referred to as is computed as the difference between the two eigenval-
octaves. The eigendecomposition of the Hessian computed ues λ̃1 , λ̃2 of the tensor field, where λ̃1 > λ̃2 . β̃(u) =  ẽ1
afterwards provides information about line-like structures. is the angle of the eigenvector ẽ1 ∈ R2 associated with the
More in detail, let u := (u, v) denote discrete pixel coor- largest eigenvalue λ̃1 , analogously to β(u).
dinates, O ∈ N the number of octaves in the pyramid, and We iterate tensor voting two times, since one pass is not
P ∈ N the number of sublevels in each octave. At the finest always sufficient [28]. Unlike Franken et al., however, we
√
resolution, we set σ to the golden ratio σ = 1 + 5/2 ≈ 1.6. At do not thin out the stickness immediately after the first pass
each octave o ∈ {0, . . . , O −1} and sublevel  ∈ {0, . . . , P − to avoid too many disconnected edges. Given the high res-
1}, we compute the Hessian by convolving the image with olution of the EL images in our dataset of approximately
the derivatives of the Gaussian kernel. To obtain the eigenval- 2500 × 2000 pixels, we use a fairly large proximity of
ues, the symmetric Hessian is diagonalized by annihilating ς1 = 15 in the first tensor voting step, and ς2 = 10 in
the off-diagonal elements using the Jacobi method which the second.
iteratively applies Givens rotations to the matrix [33]. This Figure 5d shows a typical stickness R̃(u) output. The stick-
way, its eigenvalues and the corresponding eigenvectors can ness along the orientation β̃(u) is used to extract curves at
be simultaneously extracted in a numerically stable manner. subpixel accuracy in the next step of the pipeline.
Let H = VV  denote the eigendecomposition of the Hes-
sian H, where  := diag(λ1 , λ2 ) ∈ R2×2 is a diagonal matrix
of eigenvalues λ1 > λ2 and V := (v1 , v2 ) are the associ-
ated eigenvectors. Under a Gaussian assumption, the leading 3.3 Curve extraction
eigenvector dominates the likelihood if the associated lead-
ing eigenvalue is spiked. In this sense, the local ridgeness We seek to obtain a coherent grid which we define in terms
describes the likelihood of a line segment in the image at of second-degree curves. These curves are traced along the
position u, and the orientation of the associated eigenvector previously extracted ridges by grouping centerline points by
specifies the complementary angle β(u) of the most likely their curvature. We then fit second-degree polynomials to
line segment orientation at position u. The local ridgeness these points, which yields a compact high-level curve rep-
R(u) is obtained as the maximum positive eigenvalue λ1 (u) resentation while simultaneously allowing to discard point
across all octaves and sublevels. Both the ridgeness R(u) and outliers.

123
 84 Page 6 of 23 S. Deitsch et al.

(a) (b)
Fig. 3 Extraction of ridge edges from stickness at subpixel accuracy. (a) shows a stickness patch with its initial centerline ( ) at discrete
coordinates obtained by skeletonization. The refined ridge centerline at subpixel accuracy is estimated by fitting a Gaussian function ( ) to the
cross-section profile of the ridge edge in (b) to equidistantly sampled stickness values within a predefined sampling window ( )

3
3.3.1 Extraction of ridges at subpixel accuracy

To ensure a high estimation accuracy of lens distortion 1 2

parameters, we extract ridge edges at subpixel accuracy.
This also makes the segmentation more resilient in out-of-
focus scenarios, where images may appear blurry and the Fig. 4 When considering combining two adjacent curve segments, one
−→
ridge edges more difficult to identify due to their smoother with the end line segment AB and the other with the start line segment
−−→
appearance. Blurry images can be caused by slight camera B A , we evaluate the angles α1 , α2 , and α3 and ensure they are below
the predefined threshold ϑ with α1 , α2 ≥ α3 ≥ π − ϑ. This way, the
vibrations during the long exposure time of several sec-
combined curve segments are ensured to have a consistent curvature
onds that is required for imaging. Additionally, focusing in a
dark room can be challenging, hence blur cannot be always
avoided. Nevertheless, it is beneficial to be able to operate
also on blurry images, as they can still be useful for defect form solution. On the downside, however, the method suffers
classification and power yield estimation in cell areas that do from oscillatory artifacts which require additional treatment
not irradiate. [30].
To this end, we perform non-maximum suppression by
Otsu’s global thresholding [67] on the stickness R̃(u) fol-
lowed by skeletonization [80]. Afterwards, we collect the 3.3.2 Connecting larger curve segments
points that represent the centerline of the ridges through edge
linking [48]. The discrete coordinates can then be refined by A limitation of the edge linking method is that it does not
setting the centerline to the mean of a Gaussian function fit- prioritize curve pairs with similar orientation. To address this,
ted to the edge profile [23] using the Gauss-Newton (GN) we first reduce the set of points that constitute a curve to
optimization algorithm [66]. The 1-dimensional window of a sparse representation using the nonparametric variant of
the Gaussian is empirically set to 21 pixels, with four sample the Ramer-Douglas-Peucker algorithm [21,73] introduced by
points per pixel that are computed via bilinear interpolation. Prasad et al. [71]. Afterwards, edges are disconnected if the
The GN algorithm is initialized with the sample mean and angle between the corresponding line segments is nonzero.
standard deviation in the window, and multiplicatively scaled In a second pass, two line segments are joined if they are
to the stickness magnitude at the mean. The mean of the fitted nearby, of approximately the same length, and pointing into
Gaussian is then reprojected along the edge profile oriented the same direction within an angle range ϑ = 5◦ . Figure 4
at β̃(u) to obtain the edge subpixel position. Figure 3 visu- illustrates the way two curve segments are combined.
alizes these steps. In the final step, the resulting n i points of the i-th curve
A nonparametric alternative to fitting a Gaussian to the of a line segment form a matrix Q̂(i) ∈ R2×n i . For brevity,
ridge edge profile constitutes fitting a parabola instead [19]. we denote the j-th column of Q̂(i) by q̂ j ∈ R2 . Q̂(i) is used
Such an approach is very efficient since it involves a closed- to find the parametric curve representation.

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 7 of 23 84

3.3.3 Parametric curve representation the projection on the image plane) followed by shifting the
center of distortion c := (cx , c y ) to the origin and nor-
Projected lines are represented as second-degree polyno- malizing the resulting 2-D point to the unit range using the
mials to model radial distortion. The curve parameters are dimensions M × N of the image of width M and height N .
computed via linear regression on the curve points. More Homogeneous coordinates allow to express the normaliza-
specifically, let tion conveniently using a matrix product. By defining the
upper-triangular matrix
f (x) = a2 x 2 + a1 x + a0 (1)
⎡ ⎤
sx M 0 cx
denote a second-degree polynomial in horizontal or vertical K = ⎣ 0 N cy ⎦ (3)
direction. The curve is fitted to line segment points q̂ j ∈ 0 0 1
{(x j , y j ) | j = 1, . . . , n i } ⊆ Q̂(i) of the i-th curve Q̂(i) by
minimizing the Mean Squared Error (MSE) the normalizing mapping n : Ω → [−1, 1]2 is
 
1 
ni
n(x) = π K −1 π −1 (x) , (4)
MSE( f ) = ( f (x j ) − y j )2 (2)
ni
j=1
where π : R3 → R2 projects homogeneous to inhomoge-
using RANSAC iterations [24]. In one iteration, we ran- neous coordinates,
domly sample three points to fit Eq. (1), and then determine
which of the remaining points support this curve model via 1
π : (x, y, z) → (x, y) , for z = 0 (5)
MSE. Outlier points are discarded if the squared difference z
between the point and the parabolic curve value at its posi-
tion exceeds ρ = 1.5. To keep the computational time low, and the inverse operation π −1 : R2 → R3 backprojects inho-
RANSAC is limited to 100 iterations, and stopped early once mogeneous to homogeneous coordinates:
sufficiently many inliers at a 99 % confidence level are found
[38, ch. 4.7]. After discarding the outliers, each curve is refit- π −1 : (x, y) → (x, y, 1) . (6)
ted to supporting candidate points using linear least squares
[33]. To ensure a numerically stable and statistically robust Note that the inverse mapping n−1 converts normalized
fit, the 2-D coordinates are additionally normalized [37]. image coordinates to image plane coordinates.

3.4 Curve grid model estimation 3.4.2 The field-of-view lens distortion model

The individual curves are used to jointly form a grid, which To describe the radial lens distortion, we use the first-order
allows to further discard outliers, and to estimate lens distor- Field-of-View (FOV) lens model by Devernay and Faugeras
tion. To estimate the lens distortion, we employ the plumbline that has a single distortion parameter ω. While images can
constraint [11]. The constraint models the assumption that also suffer from tangential distortion, this type of distortion
curves in the image correspond to straight lines in real world. is often negligible [92]. The sole parameter 0 < ω ≤ π
In this way, it becomes possible to estimate distortion effi- denotes the opening angle of the lens. The corresponding
ciently from a single image, which allows to use this approach radial displacement function L is defined in terms of the
also post hoc on cropped, zoomed or similarly processed distortion radius r ≥ 0 as
images.
1  ω
3.4.1 Representation of lens distortion L(r ) = arctan 2r tan , for ω = 0 . (7)
ω 2

Analogously to Devernay and Faugeras [20], we represent One advantage of the model is that its inversion has a closed-
the radial lens distortion by a function L : R≥0 → R≥0 that form solution with respect to the distortion radius r .
maps the distance of a pixel from the distortion center to a Similar to Devernay and Faugeras, we decouple the distor-
distortion factor. This factor can be used to radially displace tion from the projection onto the image plane, avoiding the
each normalized image coordinate x̃. need to calibrate for intrinsic camera parameters. Instead, the
Image coordinates are normalized by scaling down coor- distortion parameter ω is combined with the distortion cen-
dinates x := (x, y) horizontally by the distortion aspect ter c ∈ Ω and distortion aspect ratio sx which are collected
ratio sx (corresponding to image aspect ratio decoupled from in a vector θ := (c, sx , ω).

123
 84 Page 8 of 23 S. Deitsch et al.

(a) EL image of a monocrystalline PV module (b) Background-equalized image

(c) Ridgeness image ( ) from the filter responses at multiple scales (d) Stickness of ridgeness contextually enhanced using tensor voting

(e) Extracted line segments grouped by their curvature (f) Horizontal ( ) and vertical ( ) parabolic curves filtered using
the intersection constraint

Fig. 5 Visualization of the preprocessing, curve extraction, and model estimation stages for the PV module from Fig. 1

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 9 of 23 84

Normalized undistorted image coordinates x̃u = δ −1 (x̃d ) The real roots of the quartic (12) can be found with the
can be directly computed from distorted coordinates x̃d as Jenkins-Traub Rpoly algorithm [45] or a specialized quartic
solver [26]. The corresponding values f (x) are determined
L −1 (rd ) by inserting the roots back into Eq. (1).
δ −1 (x̃d ) = x̃d , for rd = 0 (8) Distortion Coefficient Estimation of the distortion coeffi-
rd
cient ω from a set of distorted image points is not straightfor-
where rd = x̃d 2 is the distance of x̃d from the origin. ward because the distortion function L(r ) is nonlinear. One
L −1 (r ) is the inverse of the lens distortion function in Eq. way to overcome this problem is to linearize L(r ) with Taylor
(7), namely polynomials, and to estimate ω with linear least squares.
To this end, we define the distortion factor
tan r ω
L −1 (r ) = , for ω = 0 . (9) L(r )
2 tan ω2 k := , for k ∈ R>0 (13)
r
The function that undistorts a point x ∈ Ω is thus
which maps undistorted image points {p j }nj=1 lying on the
  straight lines to distorted image points {q j }nj=1 lying on the
u(x) = n−1 δ −1 (n(x)) . (10) parabolic curves. Both point sets are then related by

Note that Eq. (8) exhibits a singularity at rd  0 for points pk = q . (14)

close to the distortion center. By inspecting the function’s
limits, one obtains The distorted points q j are straightforward to extract by eval-
ω uating the second-degree polynomial of the parabolic curves.
lim δ −1 (x̃d ) = ω x̃d . (11) To determine p j , we define a line with the first and the last
rd →0+ 2 tan 2 point in q j , and select points from this line. Collecting these
points in the vectors p ∈ R2n and q ∈ R2n yields an overde-
Analogously, Eq. (9) is singular at ω = 0 but approaches
termined system of 2n linear equations in one unknown. k̂ is
limr →0+ L −1 (r ) = r at the limit. In this case, Eq. (8) is
then estimated via linear least squares as
an identity transformation which does not radially displace
points.
k̂ = argmin q − pk22 , (15)
k
3.4.3 Estimation of initial lens distortion model parameters
where the solution is found via the normal equations [33] as
Lens distortion is specified by the distortion coefficient ω, the
distortion aspect ratio sx , and the distortion center c. Naive pq
k̂ := . (16)
solution leads to a non-convex objective function with sev- pp
eral local minima. Therefore, we first seek an initial set of
parameters close to the optimum, and then proceed using a The points q j , p j refer to the columns of the two matrices
convex optimization to refine the parameters. Q(i) , P(i) ∈ R2×n i , respectively, where n i again denotes the
We propose the following initialization scheme for the number of points, which are used in the following step of the
individual parameters of the FOV lens model. pipeline.
)
Distortion Aspect Ratio and Center We initialize the distor- To determine ω from the relation k = L(r r , L(r ) is
tion aspect ratio to sx = 1, and the distortion center to the expanded around ω0 = 0 using Taylor series. More specifi-
intersection of two perpendicular curves with smallest coef- cally, we use a second-order Taylor expansion to approximate
ficients in the highest order polynomial term. Such curves
can be assumed to have the smallest curvature and are thus arctan(x) = x + O(x 2 ) , (17)
located near the distortion center.
To find the intersection of two perpendicular curves, we and a sixth-order Taylor expansion to approximate
denote the coefficients of a horizontal curve by a2 , a1 , a0 ,
and the coefficients of a vertical curve by b2 , b1 , b0 . The
y3 2y 5
position x of a curve intersection is then the solution to tan(y) = y + + + O(y 6 ) . (18)
3 15
a22 b2 x 4 + 2a1 a2 b2 x 3 + x 2 2a0 a2 b2 + a12 b2 + a2 b1 + x Let L(r ) = ω1 arctan(x) with x = 2r tan(y), and y = ω2 .
·(2a0 a1 b2 + a1 b1 − 1) + a02 b2 + a0 b1 + b0 = 0 . (12) We substitute the Taylor polynomials from Eqs. (17) and

123
 84 Page 10 of 23 S. Deitsch et al.

3.4.4 Minimization criterion for the refinement of lens

distortion parameters

The Levenberg-Marquardt algorithm [50,57] is used to refine

the estimated lens distortion parameters θ. The objective
function is

1  2 (i)
n
θ  := argmin χ (P , θ ) . (21)
θ 2
i=1

P(i) ∈ R2×m is a matrix of m 2-D points of the i-th curve. The

distortion error χ 2 quantifies the deviation of the points from
the corresponding ideal straight line [20]. The undistorted
image coordinates p j := (x j , y j ) ∈ Ω are computed as
p j = u(q j ) by applying the inverse lens distortion given
in Eq. (10) to the points q j of the i-the curve Q(i) . In a
Fig. 6 Approximation of the distortion coefficient ω using Eq. (19)
( ) compared to the exact solution with respect to varying radii r .
similar manner, the obtained points p j form the columns of
For large radii outside the range of normalized coordinates (i.e., the P(i) ∈ R2×n i .
√
radius of the half-unit circle r > 1/ 2), the estimate is not accurate. This Following Devernay and Faugeras, we iteratively opti-
implies that the ideal sampled points must be both at some distance from mize the set of lens parameters θ . In every step t, we
the image border and also from the distortion center. As a side note, the
refine these parameters and then compute the overall error
estimation error becomes unacceptable for wide lenses where ω > π/4. n 2 (i)
However, the EL images in this work (ω̄) are well below this threshold t := i=1 χ (P , θ ) over all curve points. Afterwards,
we undistort the curve points and continue the optimization
until the relative change in error  := (t−1 − t )/t falls
(18), and x, y into Eq. (13) to obtain a biquadratic polyno- below the threshold  = 10−6 .
mial Q(ω) independent of r : Minimizing the objective function (21) for all parameters
simultaneously may cause the optimizer to be trapped in a
L(r ) 1 1 4 local minimum. Hence, following Devernay and Faugeras
≈ 1 + ω2 + ω . (19)
r 12 120  [20], we optimize the parameters θ = (ω, sx , c) in subsets
=:Q(ω) starting with ω only. Afterwards, we additionally optimize
the distortion center c. Finally, the parameters θ are jointly
By equating the right-hand side of Eq. (19) to k optimized.

Q(ω) = k (20) 3.4.5 Obtaining a consistent parabolic curve grid model

we can estimate ω from four roots of the resulting polyno- The layout of the curves is constrained to a grid in order to
mial Q(ω). These roots can be found by substituting z = ω2 eliminate outlier curves. Ideally, each horizontally oriented
into Eq. (19), solving the quadratic equation with respect to z, parabola should intersect each vertically oriented parabola
and substituting back to obtain ω. This eventually results in exactly once. This intersection can be found using Eq.
√
the four solutions ± z 1,2 . The solution exists only if k ≥ 1, (12). Also, every parabolic curve should not intersect other
as complex solutions are not meaningful, and thus corre- parabolic curves of same orientation within the image plane.
sponds to the largest positive real root. This set of rules eliminates most of the outliers.
We evaluated the accuracy of the approximation (19) with Robust Outlier Elimination Locally Optimized RANdom
the results shown in Fig. 6. For large radii, the approximation SAmple Consensus (LO-RANSAC) [15] is used to remove
significantly deviates from the exact solution. Consequently, outlier curves. In every LO-RANSAC iteration, the grid con-
this means that the selected points for the estimation must straints are imposed by randomly selecting two horizontal
ideally be well distributed across the image. Otherwise, the and two vertical curves to build a minimal grid model. Inliers
lens distortion parameter will be underestimated. In practice, are all curves that (1) exactly once intersect the model grid
however, this constraint does not pose an issue due to the lines of perpendicular orientation, (2) not intersect the model
spatial distribution of the solar cells across the captured EL grid lines of parallel orientation, and (3) whose MSE of the
image. reprojected undistorted points is not larger than one pixel.

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 11 of 23 84

Fig. 8 Intermediate steps of the solar mask estimation process

Δ1 Δ1 The number of curves crossing each square image area of

Δ2
solar cell is constant. Clustering the distances between the
Δ1 Δ2 curves allows to deduce the number of subdivisions within
Δ2 solar cells.
Δ1 Δ1
3.5.1 Estimation of the solar cell subdivisions and the
(a) (b) (c) number of rows and columns
Fig. 7 Estimation of solar module topology requires determining the
number of subdivisions (i.e., rectangular segments) in a solar cell. Com- The solar cells and their layout are inferred from the statis-
mon configurations include no subdivisions at all (i.e., one segment) (a), tics of the line segment lengths in horizontal and vertical
three segments (b) and four segments (c). Notice how the arrangement direction. We collect these lengths separately for each dimen-
of rectangular segments is symmetric and segment sizes increase mono- sion and cluster them. Dbscan clustering [22] is used to
tonically towards the center, i.e., Δ1 < · · · < Δn . In particular, shape
symmetry can be observed not only along the vertical axis of the solar simultaneously estimate cluster membership and the number
cell but also along the horizontal one as well of clusters. Despite the presence of outlier curves, clusters
are representative of the distribution of segment dimensions
within a cell. For example, if a solar cell consists of three
Remaining Curve Outliers Halos around the solar modules vertically arranged segments (as in Fig. 7b) with heights of
and holding mounts (such as in Fig. 5) can generate additional 20 : 60 : 20 pixels, the two largest clusters will have the
curves outside of the cells. We apply Otsu’s thresholding [67] medians 60 and 20. With the assumption that the segment
on the contrast-normalized image and discard outer curves arrangement is typically symmetric, the number of segments
that generate additional grid rows or columns with an aver- is estimated as the number of clusters times two minus one.
age intensity in the enclosed region below the automatically If clustering yields a single cluster, we assume that the solar
determined threshold. cells consist of a single segment. Outlier curves or segments,
respectively, are rejected by only considering the largest clus-
ters, with the additional constraint that the sizes of the used
3.5 Estimation of the solar module topology clusters are proportional to each other, and that not more than
two different segments (as in Fig. 7c) can be expected in a
A topology constraint on the solar cell can be employed to cell. The number of rows and columns of a solar cell is deter-
eliminate remaining non-cell curves in the background of the mined by dividing the overall size of the curve grid by the
PV module, and the number and layout of solar cells can be estimated cell side lengths.
subsequently estimated. However, outliers prevent a direct
estimation of the number of solar cell rows and columns in a 3.5.2 Curve grid outlier elimination
PV module. Additionally, the number and orientation of seg-
ments dividing each solar cell are generally unknown. Given The estimated proportions are used to generate a synthetic
the aspect ratio of solar cells in the imaged PV module, the planar grid that is registered against the curve grid intersec-
topology can be inferred from the distribution of parabolic tions. Specifically, we use the rigid point set registration of
curves. For instance, in PV modules with equally long hori- Coherent Point Drift (CPD) [64] because it is deterministic
zontal and vertical cell boundary lines, the solar cells have a and allows us to account for the proportion of outliers using
square (i.e., 1 : 1) aspect ratio. a parameter 0 ≤ w ≤ 1. We can immediately estimate w as

123
 84 Page 12 of 23 S. Deitsch et al.

the fraction of points in the synthetic planar grid and the total since the estimated curve grid already provides a global con-
number of intersections in the curve grid. text, we tackle the pixelwise classification as a combination
To ensure CPD convergence, initial positions of the syn- of adaptive thresholding and prior knowledge with regard
thetic planar grid should be sufficiently close to the curve to the straight shape of solar cells. Compared to CRFs, this
grid intersections. We therefore estimate the translation and approach does not require a training step and is easy to imple-
rotation of the planar grid to closely pre-align it with the ment.
grid we are registering against. The initial translation can be To this end, we use solar cells extracted from a PV module
estimated as the curve grid intersection point closest to the to compute a mean solar cell (see Figs. 8a, b). Since intensi-
image plane origin. The 2-D in-plane rotation is estimated ties within a mean solar cell image can exhibit a large range,
from the average differences of two consecutive intersection we apply locally adaptive thresholding [68] on 25×25 pixels
points along each curve grid row and column. This results in patches using their mean intensity, followed by a 15×15 mor-
two 2-D vectors which are approximately orthogonal to each phological opening and flood filling to close any remaining
other. The 2-D vector with the larger absolute angle is rotated holes. This leads to an initial binary mask.
by 90 % such that both vectors become roughly parallel. The Ragged edges at the contour are removed using vertical
estimated rotation is finally obtained as the average angle of and horizontal cell profiles (Fig. 8b). The profiles are com-
both vectors. puted as pixelwise median of the initial mask along each
image row or column, respectively. We combine the back-
3.5.3 Undistortion and rectification projection of these profiles with the convex hull of the binary
mask determined with the method of Barber et al. [6] to
The PV module configuration is used to undistort the whole account for cut-off corners using bitwise AND (cf., Fig. 8c).
image using Eq. (10). After eliminating the lens distortion, To further exclude repetitive patterns in the EL image of a
we use Direct Linear Transform (DLT) [38] to estimate the solar cell, e.g., due to low passivation efficiency in the con-
planar 2-D homography using the four corners of the curve tact region (see Fig. 8d), we combine the initial binary mask
grid with respect to the corners of the synthetic planar grid. and the augmented mask via bitwise XOR.
The homography is used to remove perspective distortion We note that solar cells are usually symmetric about both
from the undistorted curve grid. axes. Thus, the active solar cell area mask estimation can be
The intersections of the perspective corrected curve grid restricted to only on quadrant of the average solar cell image
may not align exactly with respect to the synthetic planar to enforce mask symmetry. Additionally, the convex hull of
grid because individual solar cells are not always accurately the solar cell and its extra geometry can approximated by
placed in a perfect grid but rather with a margin of error. polygons [1] for a more compact representation.
The remaining misalignment is therefore corrected via affine
Moving Least Squares (MLS) [81], which warps the image 3.7 Parameter tuning
using the planar grid intersections as control points distorted
using the estimated lens parameters, and curve grid intersec- The proposed solar cell segmentation pipeline relies on a
tions are used as their target positions. set of hyperparameters that directly affect the segmentation
robustness and accuracy. Table 1 provides an overview of all
3.6 Estimation of the active solar cell area parameters with their values used in this work.

We use solar cell images extracted from individual PV mod- 3.7.1 Manual search
ules to generate a mask that represents the active solar cell
area. Such masks allow to exclude the background and the Since the parameters of the proposed segmentation are intu-
busbars of a solar cell (see Fig. 8). In particular, active cell itive and easily interpretable, it is straightforward to select
area masks are useful for detection of cell cracks since they them based on the setup used for EL image acquisition.
allow to mask out the busbars, which can be incorrectly iden- Main influence factors that must be considered when
tified as cell cracks due to high similarity of their appearance choosing the parameters are image resolution and physical
[87,89]. properties of the camera lens.
Estimation of solar cell masks is related to the image label- Provided parameter values were found to work particu-
ing problem, where the goal is to classify every pixel into larly well for high resolution EL images and standard camera
several predefined classes (in our case, the background and lenses, as in our dataset (cf., Sect. 4.1). For low resolution
the active cell area). Existing approaches solve this problem EL images, however, the number of pyramid octaves and sub-
using probabilistic graphical models, such as a Conditional levels will need to be increased to avoid missing important
Random Field (CRF) which learns the mapping in a super- image details. Whereas, tensor voting proximity, on contrary,
vised manner through contextual information [40]. However, will need to be lowered, since the width of ridge edges in low

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 13 of 23 84

Table 1 Overview of segmentation pipeline parameters and their values used in this work
§ Symbol Description Used value

3.2.2 O Number of octaves in Gaussian scale-space pyramid 5

P Number of sublevels in each octave 8
σ Gaussian scale-space standard deviation 1.6
γ Gaussian scale-space pyramid downsampling factor 2
3.2.3 ν Tensor voting angular specificity 2
ς1,2 Proximity of the 1st and 2nd tensor voting steps 15, 10
3.3.1 1-D sampling window for Gaussian-based subpixel position 21
3.3.2 ϑ Maximum merge angle of two neighboring line segments 5◦
3.3.3 ρ Maximum error between fitted parabolic curve value at curve point 1.5
3.4.4  Minimal change in error during refinement of lens distortion parameters 10−6
3.5 Solar cell aspect ratio 1:1
3.6 Locally adaptive thresholding window size 25 × 25

resolution images tends to be proportional to the image reso- Sovetkin and Steland [86] on simplified masks, provide qual-
lution. This immediately affects the size of the 1-D sampling itative results and runtimes, and discuss limitations.
window for determining the Gaussian-based subpixel posi-
tion of curve points.
4.1 Dataset
Curve extraction parameters correlate with the field-of-
view of the EL camera lens. In particular for wide angle
We use a dataset consisting of 44 unique PV modules with
lenses, the merge angle ϑ must be increased.
various degrees of defects to manually select the parame-
Parabolic curve fit error ρ balances between robustness
ters for the segmentation pipeline and validate the results.
and accuracy of the segmentation result. The window size
These images served as a reference during the development
for locally adaptive thresholding used for estimation of solar
of the proposed method. The PV modules were captured in a
cell masks correlates both with the resolution of EL images,
testing laboratory setting at different orientations and using
but also with the amount of noise and texture variety in solar
varying camera settings, such as exposure time. Some of EL
cells, e.g.due to cell cracks.
images were post-processed by cropping, scaling, or rotation.
This dataset consists of 26 monocrystalline and 18 polycrys-
3.7.2 Automatic search talline solar cells. In total, these 44 solar modules consist
of 2,624 solar cells out of which 715 are definitely defec-
The parameters can also be automatically optimized in an tive with defects ranging from microcracks to completely
efficient manner using random search [7,58,74,82,83,85] or disconnected cells and mechanically induced cracks (e.g.,
Bayesian optimization [3,8,9,49,63,84] class of algorithms. electrically insulated or conducting cracks, or cell cracks due
Since this step involves supervision, pixelwise PV module to soldering [88]). 106 solar cells exhibit smaller defects that
annotations are needed. In certain cases, however, it may be are not with certainty identifiable as completely defective,
not be possible to provide such annotations because individ- and 295 solar cells feature miscellaneous surface abnormal-
ual defective PV cells can be hard to delineate, e.g., they ities that are no defects. The remaining 1,508 solar cells
appear completely dark. Also, the active solar cell area of are categorized as functional without any perceivable sur-
defective cells is not always well-defined. Therefore, we face abnormalities. The solar cells in imaged PV modules
refrained from automatically optimizing the hyperparame- have a square aspect ratio (i.e., are quadratic).
ters in this work. The average resolution of the EL images is 2779.63 ×
2087.35 pixels with a standard deviation of image width and
height of 576.42 and 198.30 pixels, respectively. The median
resolution is 3152 × 2046 pixels.
4 Evaluation Additional eight test EL images (i.e., about 15% of
the dataset) are used for the evaluation. Four modules are
We evaluate the robustness and accuracy of our approach monocrystalline and the remaining four are polycrystalline.
against manually annotated ground truth masks. Further, Their ground truth segmentation masks consist of hand-
we compare the proposed approach against the method by labeled solar cell segments. The ground truth additionally

123
 84 Page 14 of 23 S. Deitsch et al.

specifies both the rows and columns of the solar cells, and antialiasing or interpolation during mask construction, we
their subdivisions. These images show various PV modules define importance of pixels by their intensity. Given two non-
with a total of 408 solar cells. The resolution of the test EL binary masks A and B, the weighted Jaccard similarity is
images varies around 2649.50 ± 643.20 × 2074 ± 339.12 
with a median image resolution of 2581.50 × 2046. min{A(u), B(u)}
Jw =  u∈Ω . (22)
Three out of four monocrystalline modules consist of 4 × u∈Ω max{A(u), B(u)}
9 cells and the remaining monocrystalline module consists
of 6 × 10 cells. All of their cells are subdivided by busbars The performance metric is computed on pairs of segmented
into 3 × 1 segments. cells and ground truth masks. A ground truth cell mask is
The polycrystalline modules consist of 6 × 10 solar cells matched to the segmented cell with the largest intersection
each. In two of the modules, every cell is subdivided into 3 × area, thus taking structural coherence into account.
1 segments. The cells of the other two modules are subdivided We additionally compute the Jaccard index of the back-
into 4 × 1 segments. ground, which corresponds to the accuracy of the method to
segment the whole solar module. Solar cell misalignment or
4.2 Evaluation metrics missed cells will therefore penalize the segmentation accu-
racy to a high degree. Therefore, the solar module Jaccard
We use two different metrics, pixelwise scores and the index provides a summary of how well the segmentation per-
weighted Jaccard index to evaluate both the robustness and forms per EL image.
the accuracy of the proposed method and to compare our
method against related work. In the latter case, we addition- 4.3 Quantitative results
ally use a third metric, the Root Mean Square Error (RMSE),
to compute the segmentation error on simplified masks. We evaluate the segmentation accuracy and the robustness of
our approach using a fixed set of parameters as specified in
4.2.1 Root mean square error Table 1 on EL images of PV modules acquired in a material
testing laboratory.
The first performance metric is the RMSE given in pixels
between the corners of the quadrilateral mask computed from 4.3.1 Comparison to related work with simplified cell masks
the ground truth annotations and the corners estimated by the
individual modalities. The metric provides a summary of the The method by Sovetkin and Steland focuses on the esti-
method’s accuracy in absolute terms across all experiments. mation of the perspective transformation of the solar module
and the extraction of solar cells. Radial distortion is corrected
4.2.2 Pixelwise classification with a lens model of an external checkerboard calibration.
The grid structure is fitted using a priori knowledge of the
The second set of performance metrics are precision, recall, module topology. For this reason, we refer to the method as
and the F1 score [76]. These metrics are computed by Perspective-corrected Grid Alignment (PGA). The method
considering cell segmentation as a multiclass pixelwise clas- makes no specific proposal for mask generation and there-
sification into background and active area of individual solar fore yields rectangular solar cells.
cells. A typical 60 cell PV module will therefore contain up In order to perform a comparison, the exact masks (cf.,
to 61 class labels. A correctly segmented active area pixel is Fig. 9a) are restricted to quadrilateral shapes (cf., Fig. 9b).
a true positive, the remaining quantities are defined accord- The quadrilateral mask is computed as the minimum circum-
ingly. Pixelwise scores are computed globally with respect scribing polygon with four sides, i.e., a quadrilateral, using
to all the pixels. Therefore, the differences between the indi- the approach of Aggarwal et al. [1]. The quadrilateral exactly
vidual results for these scores are naturally smaller than for circumscribes the convex hull of the solar cell mask with all
metrics that are computed with respect to individual solar the quadrilateral sides flush to the convex hull.
cells, such as the Jaccard index. PGA assumes that radial distortion is corrected by an
external checkerboard calibration. This can be a limiting fac-
4.2.3 Weighted Jaccard Index tor in practice. Hence, the comparison below considers both
practical situations by running PGA on distorted images and
The third performance metric is the weighted Jaccard index on undistorted images using the distortion correction of this
[14,43], a variant of the metric widely known as Intersection- work.
over-Union (IoU). This metric extends the common Jaccard Root Mean Square Error Table 2 provides the RMSE in pix-
index by an importance weighting of the input pixels. As els between the corners of the quadrilaterals computed by
the compared masks are not strictly binary either due to the respective modality and the quadrilateral mask estimated

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 15 of 23 84

Fig. 9 Example of an exact mask (a) of solar cells estimated using the proposed approach and a quadrilateral mask (b) determined from the exact
mask. The latter is used for comparison against the method of Sovetkin and Steland [86]. Both masks are shown as color overlays. Different colors
denote different instances of solar cells

Table 2 Root Mean Square

Distorted Undistorted
Error (RMSE), in pixels, of the
distance between the corners of PGA Proposed PGA Proposed
the quadrilateral mask ω (ω, sx , c) w/ MLS ω (ω, sx , c) w/ MLS
determined from the ground
truth annotations and the corners Monocrystalline 6.09 2.71 2.61 2.64 4.00 2.88 2.68 2.61
determined by the respective Polycrystalline 5.32 2.56 2.52 2.45 2.76 2.33 1.91 1.77
method in all eight test images
Overall 5.65 2.62 2.56 2.53 3.32 2.55 2.26 2.15
Bold face denotes smallest error

from the ground truth. The metric is provided for monocrys- 4.3.2 Ablation study
talline and polycrystalline solar wafers separately, and for
both types combined. In all cases, the proposed approach out- We ablate the lens distortion parameters and the post hoc
performs both PGA variants. We particularly notice that PGA application of affine MLS to investigate their effect on the
greatly benefits from lens distortion estimation. This under- accuracy and the success rate of the segmentation process.
lines our observation that the latter is essential for highly The ablation is performed both on original (i.e., distorted)
accurate segmentation. EL images and undistorted ones.
Pixelwise Classification Pixelwise scores for the simplified Distorted vs. Undistorted EL Images For the ablation study,
masks of both methods are given in Table 3. For monocrys- we consider two main cases. In the undistorted case, both
talline PV modules, PGA generally achieves higher scores. reference and predicted masks are unwarped using estimated
However, highest scores are achieved only for images for lens distortion parameters. Then, quadrilaterals are fitted to
which the lens distortion has been removed. The proposed individual cell masks to allow a comparison against PGA
method fails to segment a row of cells in a solar mod- which always yields such quadrilateral cell masks. For a fair
ule resulting in a lower recall. However, for polycrystalline comparison, PGA is also applied to undistorted images.
PV modules, the proposed method consistently outperforms In the distorted case, however, the comparison is per-
PGA. In the overall score, the proposed method also out- formed in the original image space. Since the proposed
performs the best-case evaluation for PGA on undistorted method yields a curved grid after applying the inverse of
images. However, PGA has highest recall, which is due to lens distortion, we synthesize a regular grid from backwarped
the lower number of parameters of PGA. cell masks. Specifically, we extract the contours of estimated
Weighted Jaccard Index The Jaccard scores summarized solar cell masks to obtain the coordinates of the quadri-
as boxplots in Fig. 10 support the pixelwise classification lateral in the unwarped image, and then apply the inverse
scores, showing that the proposed method is more accurate of estimated geometric transformations to rectangle coordi-
than PGA. The latter, however, is slightly more robust. For nates. Afterwards, we fit lines to each side of the backwarped
complete modules, the considerable spread of PGA is par- quadrilaterals along grid rows and columns. From their inter-
tially attributed to one major outlier. Overall, the proposed sections we finally obtain the corner coordinates of each solar
segmentation pipeline is highly accurate. Particularly once cells in the distorted image which we can use for comparison
a cell is detected, the cell outline is accurately and robustly against distorted PGA results.
segmented. Parameterization First, we reduce the lens distortion model
to a single radial distortion parameter ω and assume both

123
 84 Page 16 of 23 S. Deitsch et al.

Table 3 Pixelwise classification

Distorted Undistorted
scores for quadrilateral masks
estimated using PGA and the PGA Proposed PGA Proposed
proposed approach ω (ω, sx , c) w/ MLS ω (ω, sx , c) w/ MLS

(a) Monocrystalline
Precision 97.55% 97.77% 99.18% 98.98% 98.43% 98.34% 99.67% 99.53%
Recall 98.37% 82.08% 97.53% 97.71% 98.87% 81.56% 96.94% 97.14%
F1 score 97.95% 86.57% 98.24% 98.24% 98.65% 86.46% 98.18% 98.21%
Accuracy 98.19% 88.43% 98.34% 98.32% 98.82% 88.55% 98.33% 98.38%
(b) Polycrystalline
Precision 97.22% 97.70% 98.82% 98.77% 98.36% 97.40% 99.52% 99.59%
Recall 97.70% 88.35% 99.29% 99.36% 99.29% 87.08% 99.17% 99.35%
F1 score 97.45% 91.32% 99.05% 99.06% 98.82% 90.42% 99.35% 99.47%
Accuracy 97.13% 92.44% 98.89% 98.90% 98.66% 92.06% 99.25% 99.39%
(c) Overall
Precision 97.37% 97.30% 99.00% 98.88% 98.38% 97.09% 99.58% 99.56%
Recall 97.78% 82.36% 98.27% 98.39% 99.01% 81.15% 97.97% 98.18%
F1 score 97.57% 87.45% 98.60% 98.60% 98.69% 86.60% 98.73% 98.83%
Accuracy 97.74% 90.15% 98.58% 98.57% 98.75% 90.06% 98.72% 98.81%
Bold face denotes the best performing method

PGA (distorted) PGA (undistorted) Proposed

(a) Solar cells (b) Modules

100 100

98
90
Jaccard index >%@
Jaccard index >%@

96

80
94

92 70

90
60

88
Mono- Poly- Overall Mono- Poly- Overall
crystalline crystalline crystalline crystalline

Fig. 10 Boxplots of Jaccard scores for the three evaluated modalities. The Jaccard scores are computed against hand-labeled ground truth masks.
In (a), the scores are computed for the individual solar cells. In (b), the scores are evaluated against the whole solar modules. The two left-most
groups in each figure correspond to boxplots with respect to different solar wafers. Whereas the right-most group summarizes the performance of
both solar wafer types combined

square aspect ratio (i.e., sx = 1) and the center of distortion high, the recall and therefore the F1 score drops consider-
to be located in the image center. During optimization, these ably (cf., Table 3). The reason for this is that such a lens
two parameters are kept constant. In this experiment, we also parametrization is too rigid. As a consequence, this weakens
do not correct the curve grid using affine MLS. The compar- the grid detection: correctly detected curves are erroneously
ison against PGA shows that such a simplistic lens model is discarded because of inaccuracies of the lens model with only
still more accurate than PGA both in the distorted and undis- a single parameter ω instead of four parameters (ω, sx , c).
torted cases (cf., Table 2). However, while the precision is

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 17 of 23 84

Table 4 Pixelwise classification scores for exact masks estimated using Solar cells Modules
the proposed approach 100
Metric Monocrystalline Polycrystalline Overall
95
Precision 97.47% 97.49% 97.53%

Jaccard index >%@

Recall 96.93% 98.90% 97.77%
F1 score 97.09% 98.19% 97.62% 90
Accuracy 97.67% 97.97% 97.80%
85

For the next comparison, we increase the number of 80

degrees of freedom. Both the distortion aspect ratio sx and the
center of distortion c are refined in addition to the radial dis- 75
tortion parameter ω. Curve grid correction via affine MLS is Mono- Poly- Overall
crystalline crystalline
again omitted. This parametrization achieves much improved
RMSE and segmentation success rates. Fig. 11 Boxplots of Jaccard scores for the proposed approach
Finally, we use the full parametrization, i.e., we refine
all lens distortion parameters (ω, sx , c) and apply post hoc
correction via affine MLS. This model is denoted as w/ MLS. a highly consistent segmentation. The IQR of whole mod-
Discussion We summarize the results of the ablation study ules is slightly larger. This is, however, not surprising since
in Tables 2 and 3. Here, w/ MLS denotes the full model that the boxplots summarize the joint segmentation scores across
includes the correction step via affine MLS. The full model multiple modules.
with post hoc affine MLS grid correction performs in many
instances best. However, applying MLS is not always ben- 4.4 Qualitative results
eficial. Particularly, for monocrystalline PV modules, grid
correction does not always improve the results. Figure 12 shows the qualitative results of the segmentation
We conclude that the proposed joint lens model estima- pipeline on four test images. The two results in the left column
tion with full parametrization and grid detection is essential are computed on monocrystalline modules, the two results in
for robustness and accuracy of the segmentation. Since the the right column on polycrystalline modules. The estimated
subsequent grid correction using affine MLS only marginally solar module curve grids are highly accurate. Even in pres-
improves the results, its application can be seen as optional. ence of complex texture intrinsic to the material, the accuracy
of the predicted solar module curve grid is not affected.
4.3.3 Segmentation performance with exact cell masks
4.5 Runtime evaluation
To allow an exact comparison of the segmentation results to
the ground truth, we inverse-warp the estimated solar cell Figure 13 breaks down the average time taken by the individ-
masks back to the original image space by using the deter- ual steps of the segmentation pipeline. Figure 14 summarizes
mined perspective projection and lens distortion parameters. the contribution of individual pipeline steps to the overall pro-
This way, the estimated solar module masks will as exactly cessing time for all 44 images. The timings were obtained
as possible overlay the hand-labeled ground truth masks. on a consumer system with an Intel i7-3770K CPU clocked
Pixelwise Classification Table 4 summarizes the pixelwise at 3.50 GHz and 32 GB of RAM. The first three stages of
classification scores for the exact masks estimated using the segmentation pipeline are implemented in C++ whereas
the proposed method. The method is more robust on poly- the last stage (except for MLS image deformation) is imple-
crystalline PV modules than on monocrystalline modules. mented in Python.
However, for both module types, the method achieves a very For this benchmark, EL images were processed sequen-
high overall accuracy beyond 97 % for all metrics. Investi- tially running only on the CPU. Note, however, that the
gation of failure cases for monocrystalline modules reveals implementation was not optimized in terms of the runtime
difficulties on cells where large gaps coincide with cell cracks and only parts of the pipeline utilize all available CPU cores.
and ragged edges. To this end, additional speedup can be achieved by running
Weighted Jaccard Index Jaccard scores for exact masks are parts of the pipeline in parallel or even on a GPU.
given in Fig. 11. The scores confirm the results of the pix- On average, it takes 1 min and 6 s to segment all solar cells
elwise metrics. Notably, the interquartile range (IQR) of in a high resolution EL image (cf., Fig. 14). Preprocessing
individual cells has a very small spread, which indicates is computationally most expensive, curve and cell extraction

123
 84 Page 18 of 23 S. Deitsch et al.

Solar cells Busbars

(a) Monocrystalline (b) Polycrystalline

Fig. 12 Qualitative segmentation results of four test images depicting the estimated curve grid superimposed over the contrast-normalized input
EL image. For visualization purposes, the original EL images were cropped

are on average cheapest. The standard deviation of the model Monocrystalline Polycrystalline
estimation step is highest (see Fig. 13), which is mostly due
30.08 s
to dependency upon the total number of ridge edges and the Pre-
28.85 s
number of resulting curves combined with the probabilistic processing
29.36 s
nature of LO-RANSAC. 8.92 s
Interestingly, processing EL images of monocrystalline Curve 9.23 s
Extraction
solar modules takes slightly longer on average than process- 9.11 s

ing polycrystalline solar modules. This is due to large gaps 23.97 s

between ridges caused by cut-off corners that produce many Model 18.83 s
Estimation
disconnected curve segments which must be merged first. 20.93 s

Conversely, curve segments in polycrystalline solar modules 6.50 s

Cell
are closer, which makes it more likely that several curve seg- Extraction
6.38 s

6.43 s
ments are combined early on.
An average processing time of 1 min and 6 s is substan- 0 10 20 30 40 50
tially faster than manual processing, which takes at least Processing time >s@
several minutes. For on-site EL measurements with in-situ
Fig. 13 Average time taken by individual steps of the segmentation
imaging of PV modules, the processing times must be fur- pipeline, in seconds. The error bars denote the upper range of the stan-
ther optimized, likely by at least a factor of ten. However, dard deviation
in other imaging environments, for example material test-
ing laboratories, the runtime is fully sufficient, given that the
handling of each module for EL measurements and the per-
formance evaluation impose much more severe scheduling
bottlenecks.

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 19 of 23 84

cells with an accuracy of 97.80 %. With respect to clas-

sification performance, the segmentation pipeline reaches
an F1 score of 97.61 %.
Additionally, we compared the proposed method against
the PV module detection approach by Sovetkin and Steland
[86], which is slightly more robust but less accurate than
our method. The comparison also shows that our joint lens
distortion estimation and grid detection approach achieves a
higher accuracy than a method that decouples both steps.
Fig. 14 Relative contribution of the average processing time for indi- Beyond the proposed applications, the method can serve
vidual pipeline steps to the overall runtime with respect to different
solar module types and both types combined
as a starting point for bootstrapping deep learning architec-
tures that could be trained end-to-end to directly segment
the solar cells. Future work may include to investigate the
4.6 Limitations required adaptations and geometric relaxations for using use
the method not only in manufacturing setting but also in the
Mounts that hold PV modules may cause spurious ridge field. Such relaxations could be achieved, for instance, by
edges. Early stages of the segmentation focus on ridges performing the grid detection end-to-end using a CNN.
without analyzing the whole image content, which may occa- Given that grid structure is pervasive in many different
sionally lead to spurious edges and eventually to an incorrect problem domains, the proposed joint lens estimation and grid
segmentation. Therefore, automatic image cropping prior to identification may also find other application fields, for exam-
PV module segmentation could help reduce segmentation ple the detection of PV modules in aerial imagery of solar
failures due to visible mounts. power plants, building facade segmentation, and checker-
While the algorithm is able to process disconnected (dark) board pattern detection for camera calibration.
cells, rows or columns with more than 50 % of disconnected
cells pose a difficulty in correctly detecting the grid due to Acknowledgements This work was funded by Energy Campus Nurem-
berg (EnCN) and partially supported by the Research Training Group
insufficient edge information. However, we observed that
1773 “Heterogeneous Image Systems” funded by the German Research
also human experts have problems to determine the contours Foundation (DFG).
under such circumstances.
We also observed that smooth edges can result in segmen- Funding Open Access funding enabled and organized by Projekt
DEAL.
tation failures. This is because the stickness of smooth edges
is weak and may completely fade away after non-maximum Open Access This article is licensed under a Creative Commons
suppression. This problem is also related to situations where Attribution 4.0 International License, which permits use, sharing, adap-
the inter-cell borders are exceptionally wide. In such cases, tation, distribution and reproduction in any medium or format, as
long as you give appropriate credit to the original author(s) and the
it is necessary to adjust the parameters of the ridgeness filter source, provide a link to the Creative Commons licence, and indi-
and the proximity of the tensor voting. cate if changes were made. The images or other third party material
in this article are included in the article’s Creative Commons licence,
unless indicated otherwise in a credit line to the material. If material
is not included in the article’s Creative Commons licence and your
5 Conclusions intended use is not permitted by statutory regulation or exceeds the
permitted use, you will need to obtain permission directly from the copy-
In this work, we presented a fully automatic segmentation right holder. To view a copy of this licence, visit http://creativecomm
method for precise extraction of solar cells from high reso- ons.org/licenses/by/4.0/.
lution EL images. The proposed segmentation is robust to
underexposure, and works robustly in presence of severe
defects on solar cells. This can be attributed to the proposed References
preprocessing and the ridgeness filtering, coupled with tensor
voting to robustly determine the inter-cell borders and bus- 1. Aggarwal, A., Chang, J.S., Yap, C.K.: Minimum area circumscrib-
ing polygons. Vis. Comput. 1(2), 112–117 (1985). https://doi.org/
bars. The segmentation is highly accurate, which allows to 10.1007/BF01898354
use its output for further inspection tasks, such as automatic 2. Ahmed, M., Farag, A.: Nonmetric calibration of camera lens dis-
classification of defective solar cells and the prediction of tortion: differential methods and robust estimation. IEEE Trans.
power loss. Image Process. 14(8), 1215–1230 (2005). https://doi.org/10.1109/
TIP.2005.846025
We evaluated the segmentation with the Jaccard index on
3. Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: a
eight different PV modules consisting of 408 hand-labeled next-generation hyperparameter optimization framework. In: Pro-
solar cells. The proposed approach is able to segment solar ceedings of the 25th ACM SIGKDD International Conference on

123
 84 Page 20 of 23 S. Deitsch et al.

Knowledge Discovery and Data Mining, Association for Comput- 21. Douglas, D.H., Peucker, T.K.: Algorithms for the reduction of the
ing Machinery, New York, NY, USA, KDD ’19, pp. 2623–2631, number of points required to represent a digitized line or its cari-
(2019) https://doi.org/10.1145/3292500.3330701 cature. Cartogr. Int. J. Geogr. Inf. Geovis. 10(2), 112–122 (1973).
4. Aloimonos, J.: Shape from texture. Biol. Cybern. 58(5), 345–360 https://doi.org/10.3138/fm57-6770-u75u-7727
(1988). https://doi.org/10.1007/BF00363944 22. Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algo-
5. Anwar, S.A., Abdullah, M.Z.: Micro-crack detection of multicrys- rithm for discovering clusters in large spatial databases with noise.
talline solar cells featuring an improved anisotropic diffusion filter In: Proceedings of the 2nd International Conference on Knowl-
and image segmentation technique. EURASIP J. Image Video edge Discovery and Data Mining, AAAI Press, Portland, OR, USA,
Process. 2014(1), 15 (2014). https://doi.org/10.1186/1687-5281- KDD’96, pp 226–231 (1996)
2014-15 23. Fabijańska, A.: A survey of subpixel edge detection methods for
6. Barber, C.B., Dobkin, D.P., Dobkin, D.P., Huhdanpaa, H.: The images of heat-emitting metal specimens. Int. J. Appl. Math. Com-
quickhull algorithm for convex hulls. ACM Trans. Math. Softw. put. Sci. 22(3), 695–710 (2012). https://doi.org/10.2478/v10006-
22(4), 469–483 (1996). https://doi.org/10.1145/235815.235821 012-0052-3
7. Bergstra, J., Bengio, Y.: Random search for hyper-parameter opti- 24. Fischler, M.A., Bolles, R.C.: Random sample consensus: a
mization. J. Mach. Learn. Res. 13, 281–305 (2012) paradigm for model fitting with applications to image analysis and
8. Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for automated cartography. Commun. ACM 24(6), 381–395 (1981).
hyper-parameter optimization. In: Shawe-Taylor, J., Zemel, R., https://doi.org/10.1145/358669.358692
Bartlett, P., Pereira, F., Weinberger, K.Q. (eds.) Advances in Neural 25. Fitzgibbon, A.: Simultaneous linear estimation of multiple view
Information Processing Systems, vol. 24, pp. 2546–2554. Curran geometry and lens distortion. IEEE Comput. Soc. Conf. Comput.
Associates, New York (2011) Vis. Pattern Recognit. 1, 125–132 (2001). https://doi.org/10.1109/
9. Bergstra, J., Yamins, D., Cox, D.D.: Making a science of model CVPR.2001.990465
search: hyperparameter optimization in hundreds of dimensions 26. Flocke, N.: Algorithm 954: an accurate and efficient cubic and quar-
for vision architectures. In: Proceedings of the 30th International tic equation solver for physical applications. ACM Trans. Math.
Conference on International Conference on Machine Learning, Softw. 41(4), 30:1-30:24 (2015). https://doi.org/10.1145/2699468
ICML’13, pp. 115–123 (2013) 27. Franken, E., van Almsick, M., Rongen, P., Florack, L., ter
10. Breitenstein, O., Bauer, J., Bothe, K., Hinken, D., Müller, J., Haar Romeny, B.: An efficient method for tensor voting using
Kwapil, W., Schubert, M.C., Warta, W.: Can luminescence imaging steerable filters. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) Euro-
replace lock-in thermography on solar cells? IEEE J. Photovolt. pean Conference on Computer Vision (ECCV), pp. 228–240 (2006)
1(2), 159–167 (2011). https://doi.org/10.1109/JPHOTOV.2011. https://doi.org/10.1007/11744085_18
2169394 28. Franken, E., Rongen, P., van Almsick, M., ter Haar Romeny,
11. Brown, D.C.: Close-range camera calibration. Photogramm. Eng. B.: Detection of electrophysiology catheters in noisy fluoroscopy
Remote Sens. 37, 855–866 (1971). https://doi.org/10.1.1.14.6358 images. In: Larsen, R., Nielsen, M., Sporring, J. (eds.) Medical
12. Buerhop-Lutz, C., Deitsch, S., Maier, A., Gallwitz, F., Berger, Image Computing and Computer-Assisted Intervention—MICCAI
S., Doll, B., Hauch, J., Camus, C., Brabec, C.J.: A benchmark 2006, Springer, Berlin, Heidelberg, pp. 25–32 (2006) https://doi.
for visual identification of defective solar cells in electrolumines- org/10.1007/11866763_4
cence imagery. In: 35th European PV Solar Energy Conference 29. Fürsattel, P., Dotenco, S., Placht, S., Balda, M., Maier, A., Riess,
and Exhibition, pp. 1287–1289 (2018) https://doi.org/10.4229/ C.: OCPAD—occluded checkerboard pattern detector. In: Winter
35thEUPVSEC20182018-5CV.3.15 Conference on Applications of Computer Vision (WACV), IEEE,
13. Chenni, R., Makhlouf, M., Kerbache, T., Bouzid, A.: A detailed (2016) https://doi.org/10.1109/WACV.2016.7477565
modeling method for photovoltaic cells. Energy 32(9), 1724–1730 30. Grompone von Gioi, R., Randall, G.: A sub-pixel edge detector: an
(2007). https://doi.org/10.1016/j.energy.2006.12.006 implementation of the Canny/Devernay algorithm. Image Process.
14. Chierichetti, F., Kumar, R., Pandey, S., Vassilvitskii, S.: Finding On Line 7, 347–372 (2017). https://doi.org/10.5201/ipol.2017.216
the Jaccard median. In: Symposium on Discrete Algorithms, pp. 31. Girshick, R.B.: Fast R-CNN. In: IEEE International Conference on
293–311, Austin, Texas (2010) Computer Vision, pp. 1440–1448 (2015), https://doi.org/10.1109/
15. Chum, O., Matas, J., Kittler, J.: Locally optimized RANSAC. In: ICCV.2015.169
Michaelis, B., Krell, G. (eds.) Pattern Recognition, vol. 2781, pp. 32. Girshick, R.B., Donahue, J., Darrell, T., Malik, J.: Rich feature hier-
236–243. Springer, Berlin (2003). https://doi.org/10.1007/978-3- archies for accurate object detection and semantic segmentation.
540-45243-0_31 In: IEEE Conference on Computer Vision and Pattern Recognition,
16. Claus, D., Fitzgibbon, A.W.: A plumbline constraint for the rational pp. 580–587 (2014). https://doi.org/10.1109/CVPR.2014.81
function lens distortion model. In: British Machine Vision Confer- 33. Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. Johns
ence (BMVC), pp. 99–108 (2005) https://doi.org/10.5244/C.19.10 Hopkins Studies in the Mathematical Sciences, Johns Hopkins Uni-
17. Claus, D., Fitzgibbon, A.W.: A rational function lens distortion versity Press (2013)
model for general cameras. Conf. Comput. Vis. Pattern Recognit. 34. Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 4th edn.
(CVPR) 1, 213–219 (2005b). https://doi.org/10.1109/CVPR.2005. Pearson, New York (2018)
43 35. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT
18. Deitsch, S., Christlein, V., Berger, S., Buerhop-Lutz, C., Maier, A., Press, Cambridge (2016)
Gallwitz, F., Riess, C.: Automatic classification of defective pho- 36. Ha, H., Perdoch, M., Alismail, H., Kweon, I.S., Sheikh, Y.: Deltille
tovoltaic module cells in electroluminescence images. Sol. Energy grids for geometric camera calibration. In: International Confer-
185, 455–468 (2019). https://doi.org/10.1016/j.solener.2019.02. ence on Computer Vision (ICCV), pp. 5354–5362 (2017). https://
067. (arXiv:1807.02894) doi.org/10.1109/ICCV.2017.571
19. Devernay, F.: A non-maxima suppression method for edge detec- 37. Harker, M., O’Leary, P., Zsombor-Murray, P.: Direct type-specific
tion with sub-pixel accuracy. Technical Report, RR-2724, INRIA conic fitting and eigenvalue bias correction. Image Vis. Comput.
(1995) https://hal.inria.fr/inria-00073970 26(3), 372–381 (2008). https://doi.org/10.1016/j.imavis.2006.12.
20. Devernay, F., Faugeras, O.: Straight lines have to be straight. 006
Mach. Vis. Appl. 13(1), 14–24 (2001). https://doi.org/10.1007/ 38. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer
PL00013269 Vision, 2nd edn. Cambridge University Press, New York (2004)

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 21 of 23 84

39. He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. 58. Masri, S.F., Bekey, G.A., Safford, F.B.: A global optimization algo-
In: IEEE International Conference on Computer Vision, pp. 2980– rithm using adaptive random search. Appl. Math. Comput. 7(4),
2988 (2017). https://doi.org/10.1109/ICCV.2017.322 353–375 (1980). https://doi.org/10.1016/0096-3003(80)90027-2
40. He, X., Zemel, R.S., Carreira-Perpinan, M.A.: Multiscale con- 59. Mauk, M.G.: Image processing for solar cell analysis, diagnostics
ditional random fields for image labeling. IEEE Conf. Comput. and quality assurance inspection. In: Image Processing: Concepts,
Vis. Pattern Recognit. 2, 695–702 (2004). https://doi.org/10.1109/ Methodologies, Tools, and Applications, IGI Global, chap 71, pp.
CVPR.2004.1315232 1426–1462 (2013)
41. Hoffmann, M., Ernst, A., Bergen, T., Hettenkofer, S., Garbas, J.U.: 60. Mayr, M., Hoffmann, M., Maier, A., Christlein, V.: Weakly super-
A robust chessboard detector for geometric camera calibration. In: vised segmentation of cracks on solar cells using normalized L p
International Joint Conference on Computer Vision, Imaging and norm. In: 2019 IEEE International Conference on Image Process-
Computer Graphics Theory and Applications (VISIGRAPP), pp. ing (ICIP), pp. 1885–1889 (2019). https://doi.org/10.1109/ICIP.
34–43 (2017). https://doi.org/10.5220/0006104300340043 2019.8803116
42. IEC TS 60904-13:2018 (2018) Photovoltaic devices—part 13: 61. Medioni, G., Tang, C.K., Lee, M.S.: Tensor voting: theory and
Electroluminescence of photovoltaic modules. Technical specifi- applications. In: Proceedings of RFIA (2000)
cation, International Electrotechnical Commission 62. Mehta, S., Azad, A.P., Chemmengath, S.A., Raykar, V., Kalya-
43. Ioffe, S.: Improved consistent sampling, weighted minhash and L 1 naraman, S.: DeepSolarEye: power loss prediction and weakly
sketching. In: International Conference on Data Mining, pp. 246– supervised soiling localization via fully convolutional networks
255 (2010). https://doi.org/10.1109/ICDM.2010.80 for solar panels. In: Winter Conference on Applications of Com-
44. Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: puter Vision (WACV), pp. 333–342 (2018). https://doi.org/10.
Spatial transformer networks. In: Cortes, C., Lawrence, N.D., Lee, 1109/WACV.2018.00043
D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Infor- 63. Močkus, J.: On Bayesian methods for seeking the extremum. In:
mation Processing Systems, 28, pp. 2017–2025. Curran Associates Marchuk, G.I. (ed.) IFIP Technical Conference on Optimization
Inc, New York (2015) Techniques, pp. 400–404. Springer, Berlin (1975)
45. Jenkins, M.A., Traub, J.F.: A three-stage algorithm for real poly- 64. Myronenko, A., Song, X.: Point set registration: coherent point
nomials using quadratic iteration. J. Numer. Anal. 7(4), 545–566 drift. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2262–2275
(1970). https://doi.org/10.1137/0707045 (2010). https://doi.org/10.1109/TPAMI.2010.46
46. Karatepe, E., Boztepe, M., Çolak, M.: Development of a suitable 65. Nian, B., Fu, Z., Wang, L., Cao, X.: Automatic detection of
model for characterizing photovoltaic arrays with shaded solar defects in solar modules: image processing in detecting. In: 2010
cells. Sol. Energy 81(8), 977–992 (2007). https://doi.org/10.1016/ International Conference on Computational Intelligence and Soft-
j.solener.2006.12.001 ware Engineering, IEEE, pp. 1–4 (2010), https://doi.org/10.1109/
47. Kaushika, N.D., Gautam, N.K.: Energy yield simulations of inter- WICOM.2010.5600703
connected solar PV arrays. IEEE Trans. Energy Convers. 18(1), 66. Nocedal, J., Wright, S.J.: Numerical Optimization, Operations
127–134 (2003). https://doi.org/10.1109/TEC.2002.805204 Research and Financial Engineering, vol. 43, 2nd edn. Springer,
48. Kovesi, P.: MATLAB and Octave functions for computer vision and New York (2006). https://doi.org/10.1007/978-0-387-40065-5
image processing (2017). http://www.peterkovesi.com/matlabfns 67. Otsu, N.: A threshold selection method from gray-level histograms.
49. Kushner, H.J.: A new method of locating the maximum point of an IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979). https://doi.org/
arbitrary multipeak curve in the presence of noise. J. Basic Eng. 10.1109/TSMC.1979.4310076
86(1), 97–106 (1964). https://doi.org/10.1115/1.3653121 68. Pitas, I.: Digital Image Processing Algorithms. Prentice-Hall Inc,
50. Levenberg, K.: A method for the solution of certain non-linear Upper Saddle River (1993)
problems in least squares. Q. Appl. Math. 2(2), 164–168 (1944) 69. Placht, S., Fürsattel, P., Mengue, E.A., Hofmann, H., Schaller,
51. Li, Y., Qi, H., Dai, J., Ji, X., Wei, Y.: Fully convolutional instance- C., Balda, M., Angelopoulou, E.: ROCHADE: robust checker-
aware semantic segmentation. In: IEEE Conference on Computer board advanced detection for camera calibration. In: Fleet, D.,
Vision and Pattern Recognition, pp. 4438–4446 (2017). https://doi. Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) European Conference
org/10.1109/CVPR.2017.472 on Computer Vision (ECCV), Lecture Notes in Computer Sci-
52. Lin, C.H., Lucey, S.: Inverse compositional spatial transformer ence, vol. 8692, pp. 766–779 (2014). https://doi.org/10.1007/978-
networks. In: IEEE Conference on Computer Vision and Pattern 3-319-10593-2_50
Recognition (CVPR), IEEE, pp. 2252–2260 (2017). https://doi. 70. Potthoff, T., Bothe, K., Eitner, U., Hinken, D., Köntges, M.:
org/10.1109/CVPR.2017.242 Detection of the voltage distribution in photovoltaic modules by
53. Lindeberg, T.: Scale-space theory: a basic tool for analyzing struc- electroluminescence imaging. Prog. Photovolt. Res. Appl. 18(2),
tures at different scales. J. Appl. Stat. 21(1–2), 225–270 (1994). 100–106 (2010). https://doi.org/10.1002/pip.941
https://doi.org/10.1080/757582976 71. Prasad, D.K., Leung, M.K.H., Quek, C., Cho, S.Y.: A novel
54. Lindeberg, T.: Edge detection and ridge detection with automatic framework for making dominant point detection methods non-
scale selection. In: Proceedings of the Conference on Computer parametric. Image Vis. Comput. 30(11), 843–859 (2012). https://
Vision and Pattern Recognition, pp. 465–470 (1996). https://doi. doi.org/10.1016/j.imavis.2012.06.010
org/10.1109/CVPR.1996.517113 72. Quaschning, V., Hanitsch, R.: Numerical simulation of current-
55. Lindeberg, T.: Edge detection and ridge detection with automatic voltage characteristics of photovoltaic systems with shaded solar
scale selection. Int. J. Comput. Vis. 30(2), 117–156 (1998). https:// cells. Sol. Energy 56(6), 513–520 (1996). https://doi.org/10.1016/
doi.org/10.1023/A:1008097225773 0038-092X(96)00006-0
56. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks 73. Ramer, U.: An iterative procedure for the polygonal approximation
for semantic segmentation. In: Conference on Computer Vision of plane curves. Comput. Graph. Image Process. 1(3), 244–256
and Pattern Recognition (CVPR), pp. 3431–3440 (2015). https:// (1972). https://doi.org/10.1016/S0146-664X(72)80017-0
doi.org/10.1109/CVPR.2015.7298965 74. Rastrigin, L.A.: The convergence of the random search method in
57. Marquardt, D.W.: An algorithm for least-squares estimation of non- the extremal control of a many parameter system. Autom. Remote
linear parameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1963). Control 24, 1337–1342 (1963)
https://doi.org/10.1137/0111030 75. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-
time object detection with region proposal networks. IEEE Trans.

123
 84 Page 22 of 23 S. Deitsch et al.

Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017). https://doi. 92. Tsai, R.Y.: A versatile camera calibration technique for high-
org/10.1109/TPAMI.2016.2577031 accuracy 3D machine vision metrology using off-the-shelf TV
76. van Rijsbergen, C.J.: Information Retrieval, 2nd edn. Butterworth- cameras and lenses. IEEE J. Robot. Autom. 3(4), 323–344 (1987).
Heinemann, Oxford (1979) https://doi.org/10.1109/JRA.1987.1087109
77. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional net- 93. Tseng, D.C., Liu, Y.S., Chou, C.M.: Automatic finger interrup-
works for biomedical image segmentation. In: Navab, N., Horneg- tion detection in electroluminescence images of multicrystalline
ger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing solar cells. Math. Probl. Eng. 2015, 1–12 (2015). https://doi.org/
and Computer-Assisted Intervention—MICCAI 2015, pp. 234– 10.1155/2015/879675
241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-
24574-4_28
78. Rosten, E., Loveland, R.: Camera distortion self-calibration using
Publisher’s Note Springer Nature remains neutral with regard to juris-
the plumb-line constraint and minimal Hough entropy. Mach. Vis.
dictional claims in published maps and institutional affiliations.
Appl. 22(1), 77–85 (2011). https://doi.org/10.1007/s00138-009-
0196-9. (arXiv:0810.4426)
79. Rufli, M., Scaramuzza, D., Siegwart, R.: Automatic detection of
checkerboards on blurred and distorted images. In: International
Conference on Intelligent Robots and Systems, pp. 3121–3126 Sergiu Deitsch received a masters degree in computer science from
(2008). https://doi.org/10.1109/IROS.2008.4650703 Nuremberg Institute of Technology in 2011. He is currently pursuing
80. Saeed, K., Tabȩdzki, M., Rybnik, M., Adamski, M.: K3M: a uni- a Ph.D. degree with the Pattern Recognition Lab at the Friedrich-
versal algorithm for image skeletonization and a review of thinning Alexander University Erlangen-Nürnberg. His research interests
techniques. Int. J. Appl. Math. Comput. Sci. 20(2), 317–335 (2010). include various aspects of image processing, object detection and track-
https://doi.org/10.2478/v10006-010-0024-4 ing, sensor fusion, and machine learning.
81. Schaefer, S., McPhail, T., Warren, J.: Image deformation using
moving least squares. ACM Trans. Graph. 25(3), 533 (2006).
https://doi.org/10.1145/1141911.1141920 Claudia Buerhop-Lutz has been working as a scientist in the field of
82. Schrack, G., Choit, M.: Optimized relative step size random quality control of photovoltaics for more than 15 years. She is spe-
searches. Math. Program. 10(1), 230–244 (1976). https://doi.org/ cialized in using imaging techniques, such as electroluminescence and
10.1007/BF01580669 thermography, for defect detection in solar modules.
83. Schumer, M., Steiglitz, K.: Adaptive step size random search. IEEE
Trans. Autom. Control 13(3), 270–276 (1968). https://doi.org/10.
1109/TAC.1968.1098903
84. Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian opti- Evgenii Sovetkin graduated in mathematics in 2013 and received his
mization of machine learning algorithms. In: Pereira, F., Burges, Ph.D. degree at RWTH University Aachen in 2020. From 2019, he
C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural is working in IEK-5, Institute of Photovoltaics at Forschungszentrum
Information Processing Systems, vol. 25, pp. 2951–2959. Curran Jülich. His research interests include statistical modeling for high-
Associates Inc, New York (2012) dimensional data analysis and statistical modeling for physics.
85. Solis, F.J., Wets, R.J.B.: Minimization by random search tech-
niques. Math. Oper. Res. 6(1), 19–30 (1981). https://doi.org/10.
1287/moor.6.1.19 Ansgar Steland received M.Sc. and Ph.D. degrees in mathematics
86. Sovetkin, E., Steland, A.: Automatic processing and solar cell from the Georg-August University of Göttingen, Germany. He held
detection in photovoltaic electroluminescence images. Integr. positions as an assistant at the Technical University of Berlin, Berlin,
Comput. Aided Eng. 26(2), 123–137 (2019). https://doi.org/10. Germany, as a consultant in industry, as a postdoc at the European
3233/ICA-180588 University Viadrina of Frankfurt/Oder, Germany, and as a lecturer at
87. Spataru, S., Hacke, P., Sera, D.: Automatic detection and evalua- the Faculty of Mathematics at the Ruhr-University Bochum. Since
tion of solar cell micro-cracks in electroluminescence images using 2006, he is Full Professor at RWTH Aachen University, Germany,
matched filters. In: 2016 IEEE 43rd Photovoltaic Specialists Con- where he holds the Chair of Stochastics at the Institute of Statis-
ference (PVSC), pp. 1602–1607 (2016). https://doi.org/10.1109/ tics. He is an Elected Member of the International Statistical Insti-
PVSC.2016.7749891 tute (ISI), acts as the speaker of the Society for Reliability, Quality
88. Spertino, F., Ciocia, A., Leo, P.D., Tommasini, R., Berardone, I., and Safety, and chairs the Section on Statistics in Natural Sciences
Corrado, M., Infuso, A., Paggi, M.: A power and energy procedure and Technology of the German Statistical Society (DStatG). He orga-
in operating photovoltaic systems to quantify the losses according nized and co-organized several international workshops and invited
to the causes. Sol. Energy 118, 313–326 (2015). https://doi.org/10. sessions, edited several proceedings and special journal issues, and
1016/j.solener.2015.05.033 serves the community as associate editor. His current research interests
89. Stromer, D., Vetter, A., Oezkan, H.C., Probst, C., Maier, A.: are in nonparametric statistics and high-dimensional inference, statis-
Enhanced crack segmentation (eCS): a reference algorithm for seg- tical and machine learning, neural networks, change-point detection
menting cracks in multicrystalline silicon solar cells. IEEE J. Pho- and sequential analysis, image processing, and applications in photo-
tovolt. 9(3), 752–758 (2019). https://doi.org/10.1109/JPHOTOV. voltaics, quality control and econometrics.
2019.2895808
90. Tsai, D.M., Wu, S.C., Li, W.C.: Defect detection of solar cells in
electroluminescence images using Fourier image reconstruction.
Sol. Energy Mater. Sol. Cells 99, 250–262 (2012). https://doi.org/
10.1016/j.solmat.2011.12.007
91. Tsai, D.M., Wu, S.C., Chiu, W.Y.: Defect detection in solar modules
using ICA basis images. IEEE Trans. Ind. Inform. 9(1), 122–131
(2013). https://doi.org/10.1109/TII.2012.2209663

123
Segmentation of photovoltaic module cells in uncalibrated electroluminescence... Page 23 of 23 84

Andreas Maier received his Ph.D. in 2009. From 2005 to 2009, he Christian Riess received the Ph.D. degree in computer science from
was working at the Pattern Recognition Lab at the Computer Sci- the Friedrich-Alexander University Erlangen-Nürnberg (FAU), Erlan-
ence Department of the University of Erlangen-Nuremberg. His major gen, Germany, in 2012. From 2013 to 2015, he was a Postdoc at
research subject was medical signal processing in speech data. From the Radiological Sciences Laboratory, Stanford University, Stanford,
2009 to 2010, he started working on flat-panel C-arm CT as post- CA, USA. Since 2015, he is the head of the Phase-Contrast X-ray
doctoral fellow at the Radiological Sciences Laboratory in the Depart- Group at the Pattern Recognition Laboratory at FAU. Since 2016, he is
ment of Radiology at the Stanford University. From 2011 to 2012, senior researcher and head of the Multimedia Security Group at the IT
he joined Siemens Healthcare as innovation project manager and was Infrastructures Lab at FAU, and he received his habilitation from FAU
responsible for reconstruction topics in the Angiography and X-ray in 2020. He served on the IEEE Information Forensics and Security
Business Unit. In 2012, he returned the University of Erlangen- Technical Committee 2017-2019 and received the IEEE Signal Pro-
Nuremberg as head of the Medical Reconstruction Group at the Pat- cessing Award in 2017. His research interests include all aspects of
tern Recognition Lab. In 2015, he became professor and head of the image processing and imaging, particularly with applications in image
Pattern Recognition Lab. and video forensics, X-ray phase contrast, and computer vision.

Florian Gallwitz graduated in computer science in 1994 and received

his Ph.D. degree in computer science at the Friedrich-Alexander Uni-
versity Erlangen-Nürnberg, Erlangen, Germany, in 2001. From 2000
to 2009, he worked in the speech technology industry, as a co-founder
and head of R&D of the Sympalog Voice Solutions GmbH. In 2010,
he was appointed as a Professor for computer science and media at the
Nuremberg Institute of Technology. His research interests include pat-
tern recognition, speech and image processing, and deep learning.

123