IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 15, NO.

3, JULY 2018 1205

Efficient Task and Path Planning for Maintenance

Automation Using a Robot System
Christian Friedrich , Akos Csiszar, Armin Lechler, and Alexander Verl

Abstract— The research and development of intelligent

automation solutions is a ground-breaking point for the factory
of the future. A promising and challenging mission is the use
of autonomous robot systems to automate tasks in the field of
maintenance. For this purpose, the robot system must be able
to plan autonomously the different manipulation tasks and the
corresponding paths. Basic requirements are the development
of algorithms with a low computational complexity and the
possibility to deal with environmental uncertainties. In this paper,
an approach is presented, which is especially suited to solve the
problem of maintenance automation. For this purpose, offline
data from CAD is combined with online data from an RGBD
vision system via a probabilistic filter, to compensate uncertainties
from offline data. For planning the different tasks, a method
is explained, which uses a symbolic description, founded on a
novel sampling-based method to compute the disassembly space.
Fig. 1. General architecture of the autonomous maintenance robot.
For path planning, we use global state-of-the-art algorithms
with a method that allows the adaption of the exploration
stepsize in order to reduce the planning time. Every method maximizing the factory’s economics. Abele et al. [1] show
is experimentally validated and discussed. that the maintenance costs amount up to 35% of the total life
cycle costs for machines tools. For supporting the maintenance
Note to Practitioners—This paper was motivated by the current
deficit to automate maintenance tasks in manufacturing systems.
staff and improving the factories economics, the development
Today there exist many automatic methods for fault detection, of automation solutions is a key factor.
but there is no possibility to restore the desired state. A typical While the use of automation solutions has been extensively
application is the replacement of a defective component or service discussed for fault diagnosis [2], [3], there is a technology
tasks such as refilling of cooling lubricant in machine tools. The gap in systems for restoring the intended plant or machine
focus of this paper is to provide automatic planning concepts
for robot systems which deal with this problem. For this, a task
states. For the automation of repair or service tasks, the use
planning algorithm is proposed which uses CAD and vision data. of robotic systems is common, especially in areas inaccessible
The advantage of this method is that it works on a contact analy- for humans. However, these are developed mostly for specific
sis, based on a discrete approximation of the disassembly space, areas, such as energy transmission networks [4], [5], sewer
allowing a fast determination of possible robot manipulations. systems [6], [7], or thermal boilers [8]. Also, for complex areas
Also, a new preprocessing concept for path planning is developed
which enables the adaptation of the stepsize of the path planner,
such as bridges [9], [10], material flow systems [11], or process
based on the local occupancy of the environment. Through this, plants [12], [13], the use of robot systems was investigated.
it is possible to drastically reduce the required planning effort. However, the state of the art discusses no generalized approach
Index Terms— Intelligent robots, maintenance, manufacturing to plan and execute different manipulation tasks for mainte-
automation. nance automation. Typical applications are the exchange of
broken components in a manufacturing system or different
I. I NTRODUCTION service tasks such as cleaning. Fig. 1 shows the used system

I N TODAY’S industrial context, the availability of manu-

facturing systems plays an increasingly important role in
for this paper.
The goal of this paper is to present a fundamental approach
for task and path planning, in order to enable the auto-
Manuscript received February 6, 2017; revised June 8, 2017; accepted
October 1, 2017. Date of publication December 4, 2017; date of current ver- matic execution of maintenance tasks by a robotic system.
sion July 2, 2018. This paper was recommended for publication by Associate Furthermore, we propose a method for combining the
Editor D. Liu and Editor H. Ding upon evaluation of the reviewers’ comments. symbolic-oriented task with the path planning component.
This work was supported by the German Research Foundation under
Grant DFG GSaME F2-004. (Corresponding author: Christian Friedrich.)
The authors are with the Institute of Control Engineering of Machine Tools II. S TATE - OF - THE -A RT
and Manufacturing Units, University Stuttgart, 70174 Stuttgart, Germany
(e-mail: [email protected]; akos.csiszar@isw. In this section, we will discuss the most relevant work in
uni-stuttgart.de; [email protected]; alexander.verl@isw. related domains like (dis-) assembly automation. For a general
uni-stuttgart.de). overview see [14] and [15].
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org. The field of assembly automation has a long tradition in
Digital Object Identifier 10.1109/TASE.2017.2759814 robotics. To model mechanical assemblies, usually methods
1545-5955 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
1206 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 15, NO. 3, JULY 2018

from graph theory are used, where vertices are the components planning methods such as A∗ [37] and anytime algorithms
and edges are the connection between them. Commonly used such as ARA∗ [38] are complete and optimal if a solution
representations are the Liaison-Graph [16] or more extended exists [39]. Sampling-based planning such as probabilistic
methods such as the relation graph [17], which is also used roadmaps or rapidly-exploring random trees (RRT) and more
in this approach. Through the so-called symbolic spatial extended variants [40], [41] are often better suited for high-
relations (SSR) [18] which represent the contact between dimensional problems and are probabilistically complete [35].
components, the symbolic degree of freedom (DoF) can be A global approach is also considered in this paper, because
principally determined. during the execution of the different tasks a static environment
To model disassembly or assembly sequences there is assumed. In addition, it makes it easier to combine task and
are different approaches, which use directed graphs, path planning through the used environment model. However,
AND / OR-graphs [19], diamond diagrams [20], or petri a big problem of global path planning is that the computa-
nets [21]. De Mello and Sanderson [22] discuss the theoretical tional time scales differ depending on the occupancy of the
aspects in relation to completeness and correctness of the environment. In order to deal with this problem, a method
different representations. In general, the calculation of an opti- is required to adapt the exploration stepsize of the path
mal disassembly sequence is NP-complete [23], which places planning algorithms automatically for reducing the planning
high demands on the design of appropriate heuristics [24]. time and increasing the success rate for finding a possible
For computing a possible sequence, different methods are path. To provide a general solution in maintenance automa-
known in the literature, such as the directional and nondi- tion, the developed methods should satisfy the following
rectional blocking graph [25], the disassembly precedence requirements.
matrix [21], [23], or the cut-set method [26]. The cut-set 1) Task and path planning algorithms, with a low computa-
method decomposes the connection graph into all possible tional complexity to find the required manipulations for
subgraphs, which is in general an assembly-by-disassembly solving a maintenance task.
strategy. This approach is still the basis for many advanced 2) Fusion of passive (CAD data) and active informa-
and new methods. tion (vision data) in order to incorporate environmental
One of the most sophisticated assembly planning systems is uncertainties.
High Lap (high-level assembly planning) [17], [27], [28]. The 3) Coupling of the symbolic task planning with the numer-
system combines planning and execution of the assembly tasks ical path planning into a general control concept.
by means of a robot system and uses for this purpose CAD
data and user assistance. Thomas [24], Thomas et al. [29], III. AUTOMATIC TASK AND PATH P LANNING
and Thomas and Wahl [30], extend the system by using more FOR M AINTENANCE AUTOMATION
efficient algorithms to solve the geometric decomposition. For the automation of maintenance tasks through a robot
These methods are extremely suitable for assembly automa- system the robot must be able to autonomously plan the differ-
tion, because the computation of a nearly optimal assembly ent tasks and the corresponding paths. Therefore, we suggest
sequence is provided. A disadvantage of this method for novel solutions for task and path planning, which are adapted
maintenance automation is that the planning is based purely to the specific requirements mentioned in Section II. Further,
on geometric CAD data, which makes it time-consuming and we provide a general solution for compensating environmental
unable to compensate deviations from the real environment. uncertainties via probabilistic filtering and a method to inter-
Another problem is that the task is directly decomposed into face the symbolic task with the path planning in a general
skill primitives [31], so that only prespecified tasks can be control scheme.
solved. The use of more symbolic definitions of robot skills
above the skill primitive layer could solve this problem.
A. Task Planning
Solutions in disassembly automation, which are used for
recycling tasks, usually combine real environment information Previous work of the task planning approach was also
from vision with offline available product data, compare, discussed in [42], and is here only explained for a better under-
[32], [33], and [34]. A drawback of all these solutions is that standing of the complete system. For the design of the task
they do not provide an embedded scheme for sequence and planner, we use an extended version of the relational assembly
path planning. In addition, they discuss no general method of model RM [17]. The relational assembly model contains
combining the offline with the online data from vision and information about the SSR ∈ {concentric, congruent, screwed}
also provide no solution in regards to handling classification between two components C, which are defined on a feature
errors caused by vision algorithms. geometry F G SSR ∈ {plane, line, point, circle}. For all SSR
After task planning, the corresponding path must be also a feature vector f SSR = [oSSR eSSR]T is available
determined. In principle, the topic of path planning is with an origin oSSR = [ o x o y oz ]T and a normal vector
regularly discussed in robotics and there exist several mono- eSSR = [ ex e y ez ]T at the origin. The relations are stored
graphies [35], [36], which cover the topic. The methods in a graph GSSR(C, SSR). Therefore, the required data from
can be divided into local, global, and hybrid path plan- the CAD assembly model is imported into the planner. Further-
ning approaches. Global approaches, which are especially more, the relation graph is generated using a new sampling-
well suited for our approach, use search- or sampling- based approach to compute the relative DoFs between all
based methods to find a collision-free path. Search-based components. The resulting graph can be further used to plan

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
FRIEDRICH et al.: EFFICIENT TASK AND PATH PLANNING FOR MAINTENANCE AUTOMATION USING A ROBOT SYSTEM 1207

Algorithm 1 : Sampling-Based DoF Analysis (SDA)

Input: relational assembly model GSSR(C, SSR).
Output: disassembly space WD,C ; relation graph
G (C,

). 

1: // Compute sampled sphere with N-points

2: S ← marsagliaSphere(n)
3: W D,i ← S;
4: for i = 1 : 1 : |C|
5: for j = 1 : 1 : |F G SSR|  
6: W D, j ← getSubSpace F G SSR, j , f SSR ; Fig. 2. Sampling-based DoF analysis. (i) Sampling the sphere to get the
7: end for initial disassembly space W D without any defined relation. Sampling of the
8: W D,i ← intersect(W D, j , W D,i ); feature geometries (ii) FG SSR,1 and (iii) FG SSR,2 to get the disassembly
space of the defined SSR. (iv) Computation of the resulting disassembly space
9: W D,C  ← W D,i ; of component C1 using sort and compare operations.
10: end for
11: G ← getRelationalGraph(W D,C , GSSR);



 W D (F G SSR, j ), ∀ j is computed. For example, the disassembly

12: return W D,C , G ; space of a congruent connection on a plane is described as a


hemisphere. The computation is done using the parameteriza-

1: function (S) ← marsagliaSphere(n) tion of the feature geometry, which is further rotated in the
2: i = 1; direction of the feature vector f SSR. To compute the resulting
3: while (i ≤ n) disassembly space of C j (1) is considered
4: // Get two random numbers from a uniform distribution  
5: ω1 ← U(−1, 1); M
W D,i = W D (F G SSR, j ) ∩ W D . (1)
6: ω2 ← U(−1, 1); j =1
7: if (ω12 + ω22 <1) then
The evaluation of (1) becomes straightforward. Because all
8: x(i ) ← 2ω1 1 − ω12 − ω22 ; geometric elements are represented as a discrete set, the values

9: y(i ) ← 2ω2 1 − ω12 + ω22 ; are sorted and compared to get the intersection set. So the com-
plexity of computing W D,i is in O(n · log(n)). The symbolic
10: z(i ) ← 1 − 2(ω12 + ω22 );
relation is determined based on the resulting disassembly
i = i + 1;


11:
space of the component. Algorithm 1 and Fig. 2 summarize
12: end if
this approach.
13: end while
After the computation of the relation graph G (C, ),
S ← [x(:) y(:) z(:)];


14: 

15: end function the hybrid manipulation planner computes the different manip-
ulations using a symbolic description directly in the robot
task space. The common goal is to disassemble a broken
the required actions based on the concept of manipulation component Cda , so that G ∩ Cda := ∅. For this purpose,


primitives. This results in a set of topological executable

we introduce the concept of manipulation primitives MP.
manipulation primitives, which are further sequenced with
Here, manipulation primitives are a symbolic description,
respect to the absolute DoF of a component.
which describes the skills of the robot. The skills are defined
1) Hybrid Manipulation Planning: The hybrid manipula-
by the following concepts:
tion planner (HMP) is divided into a preprocessor to com-
move(·): Describes a movement from the symbolic pose i to
pute the relation graph G


and in a processor to find


the symbolic pose i+1 in the reference frame FR.


the required set of manipulations {MP} to solve a task. twist(·): Describes a rotatory movement from the symbolic
The relation graph defines the components C as vertices pose i to the symbolic pose i+1 in the reference
 

and the edges between them as the symbolic DoF ∈ 

frame FR, with a required symbolic force/torque . 

{fix, lin, rot, fits, agpp, free} [17]. To compute the relation pull(·): Describes a translational movement from the sym-
i
graph G we introduce a new concept, based on a sampling-

bolic pose i to the symbolic pose i+1 in the
 

based geometric analysis. A sampling-based approach is con- reference frame FR, with a required symbolic
sidered, because it allows a fast computation of the possible force/torque . 

i
disassembly space W D . Therefore, a unit sphere S = {x ∈ R3 | put(·): Describes a translational movement from the sym-
x2 = 1} is computed in a discrete manner using the bolic pose i to the symbolic pose i+1 in the
 

Marsaglia method [43]. The sphere S represents the complete reference frame FR, with a required symbolic
disassembly space W D of all translational disassembly direc- force/torque .
i


tions for an unconnected component. Further, for every feature To compute the required manipulation primitives and to
geometry F G SSR, j which is connected from a component solve the task, a so-called symbolic disassembly logic is
Ci to other components, the disassembly space W D,i = introduced. The symbolic disassembly logic describes the state

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
1208 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 15, NO. 3, JULY 2018

TABLE I
E VALUATED M ETRICS FOR TASK P LANNING

Fig. 3. Preprocessing approach: CAD input data ( top left); resulting disas-
sembly direction in which the components can be moved in the disassembly
space for the base ( top right), block 2 (down left) and block 1 (down right).

transitions δ : Z ×  → Z, for a set of states Z :=

(Ci ), ∀i ∈ G
 , which is dependent upon the applied
j 

actions  := {MP} ∈ {move, twist, pull, put}. So the state

transition is given by
an Intel Core i5-6200U, 2.3 GHz with 8GB RAM. The metrics
 (Ci ) := δ (  (Ci ), MPk ). (2) used are:
k+1 k

To apply a manipulation primitive to a component, also a tool 1) preprocessing time tpre in [s] per component;
τ j is required. So we can rewrite (2) and define it by the 2) processing time tpro in [s] per component; and
rule (3) that the state (Ci ) is changed to
 (Ci ) if 
3) set of required manipulation primitives {MP}.
k k+1
a manipulation primitive MPk is applied to Ci with a tool τ j |C| represents the amount of components and | | the


number of connections per assembly. Since no collision analy-

 (Ci ) ⇐ MPk [τ j ,  (Ci )]. (3) sis is required to compute the disassembly direction of the
k+1 k
components, the planning can be rapidly done using only
To update the resulting , the processor uses a knowl-

the contact information and consider only the relative DoF.
k+1
edge base with basic patterns or sends a query to the pre- The task planning is further done in a purely symbolic way to
processor to update G . A rule-based inference is used for

generate the required manipulations.
The average planning time is 0.05 s for the computation of a
the choice of the manipulation primitive (4) which depends
manipulation primitive (pre and processing). The disadvantage
on the actual state of the component and also to derive the
of this method is, of course, that the information about the
tool (5)
contact states is required. Another drawback at this moment is
 (Ci ) ⇒ MP j (4) that we can only generate linear monotonous plans. If a single
k component has no relative DoF, the planner cannot combine
Ci ∧ MP j ⇒ τ j . (5)
different components to a subassembly to find a possible
The result is a set of manipulation primitives {MP}, which disassembly space. Also, for the use of a dual-arm kinematics
are only executable if the relative disassembly space is equal the planning algorithm must be extended to a broader planning
to the absolute, because the planner respects only the relative class.
DoF. To sequence the set in an executable way, Section III-B
introduces a method based on visibility and accessibility of B. Data Fusion and Metric Assignment
the components. To fuse the passive information from CAD with the
2) Experimental Results: Task Planning: The HMP is eval- active information from vision, we apply a probabilistic
uated using different academic and application-oriented exam- approach. This provides a way to fuse the information and to
ples. In Fig. 3 the preprocessing approach is shown using compensate for the environmental uncertainty. Based on this
Algorithm 1. After sampling the sphere for every component, a possibility is shown, which allows time efficient sequencing
the disassembly space is computed. Table I shows the results of the given set of manipulation primitives {MP}, which is
for different assembly groups. All experiments are done using evaluated using different heuristics.

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
FRIEDRICH et al.: EFFICIENT TASK AND PATH PLANNING FOR MAINTENANCE AUTOMATION USING A ROBOT SYSTEM 1209

of the manipulation primitives has to be optimized for mini-

mum overall execution time, as presented in the following:
t ∗ = arg min MP. (8)
tpath ,tτ ∈D

As it can be seen in Fig. 4, the manipulation primitives

can be represented as a graph structure and the transition
times (travel time tpath and tool change time tτ ) can be
represented as edges costs in the graphs. The problem with
Fig. 4. Concept of visibility and accessibility space for task sequencing.
this solution is similar to a traveling salesman problem (TSP)
for which many solutions are well studied. The approach
selected here is based on the A∗ algorithm, since it is already
1) Probabilistic Information Fusion: To update the degree
used for path planning in this project. A∗ is not the most
of belief about the environment information, we apply a
typical approach for solving TSP problems, and so it might
probabilistic model P(Ci |z t ). So the computation of the degree
not be the most efficient way but it is effective (returns the
of belief bel(Ci,t ) that a component Ci is on the position r C,i
optimal results). Furthermore, the proposed concept is not
can be updated using a binary Bayes-Filter (6)
dependent on one given optimization method, this is why it
bel(C i,t ) SCi bel(C i,t−1 ) was interesting to explore the differences between different
   types of heuristics and also highly efficient methods (such as
P(Ci |z 1:t ) P(Ci |z t ) P(Ci |z 1:t −1 ) the nearest-neighbor (NN) method), which do not guarantee an
= · . (6)
1 − P(Ci |z 1:t ) 1 − P(Ci |z t ) 1 − P(Ci |z 1:t −1 ) optimal solution. In order to use A∗ for sequencing, the stop
condition has been changed. Instead of reaching a goal state,
The initial degree of believe is defined with bel(Ci,0 ) =
the stop condition is reaching all required states. The only way
bel(Ci− ). For each component a separate binary Bayes-Filter
the runtime of the A∗ algorithm can be significantly improved
is allocated, so
is through the use of heuristics. To compare the efficiency,
⎛ ⎞ ⎛ ⎞ the following three heuristics have been used.
bel(C0,t ) bel(C0,t −1 )
⎜bel(C1,t )⎟ ⎜bel(C1,t −1 )⎟ 1) Zero Heuristics (Dijkstra): Using a constant zero heuris-
⎜ ⎟ ⎜ ⎟ tic the A∗ algorithm is transformed in the Dijkstra
⎜ .. ⎟ = diag(SC0 , SC1 , . . . , SC N ) · ⎜ .. ⎟. (7)
⎝ . ⎠ ⎝ . ⎠ algorithm. The predicted cost of reaching the target is
bel(C N,t ) bel(C N,t −1 ) constantly zero.
2) Nearest (Unvisited) Neighbor Heuristics (A∗ -NN):
For a defined threshold bel(Ci,t ) ≥ Ptrust,i , the information In this second case, the predicted cost for reaching the
state is assumed to be the truth and is accepted by the goal is equal to the cost of transitioning to the NN node
environment model. in the graph, which has previously not been visited in
2) Visibility and Accessibility Space: To compute an the solution candidate.
ordered sequence of manipulation primitives MP from the 3) Minimum Spanning Tree Heuristics (A∗ -MST): The
set {MP}, we introduce the visibility space WV and the third heuristics implemented for comparison of running
accessibility space W A . This means that all components Ci , times is based on selecting the total cost of the minimum
which are visible at the actual disassembly state, can be subtree, formed in between the nodes of the graph not
manipulated, because there exists a subset {MPvis } ⊆ {MP}. yet visited in the candidate solution.
The visibility of Ci results from the states of (7). Further, Also a basic NN algorithm is applied.
it is tested whether the components are accessible at the 1) Experimental Results (Task Sequencing): The task
actual disassembly state, which results in a further subset sequencing has been carried out for a sample set of four, five,
{MPacc } ⊆ {MPvis} ⊆ {MP}. The resulting subset {MPacc } and six manipulation primitives. All experiments are repeated
can be further used for task sequencing, compare Fig. 4. 100 times with an initialized random cost matrix. Fig. 6 shows
This method allows only to consider a subset of {MP} the results of the task sequencing process.
and not to find the global optimal sequence. But with this Using the A∗ with valid heuristics guarantees an optimal
method it is possible to rapidly compute a local optimal solution but has bad scaling of the planning time. On the
sequence over W A , because only one collision analysis for other hand the basic NN algorithm scales well with the
the interaction of the robot with the component is required. amount of manipulation primitives but guarantees no optimal
solution. The improvement of the task executing time t ∗ using
A∗ with heuristics is in average approximately 21% against
C. Task Sequencing
the use of the NN solution, compare Fig. 6 (right). The
After the manipulation planning a list of manipulation disadvantage of the optimal solution is of course the scaling
primitives are available. All of these have to be carried out of the planning time. Using four manipulation primitives in
by the robot in order to complete the task. However, the order contrast to six results in this case using Dijkstra in a 9.3 times
in which these are carried out is not strictly defined. In order to higher planning time t p . Using A∗ -NN results in a 46.8 times
assure the efficiency of the maintenance process, the sequence and for A∗ -MST in a 10.2 times higher planning time,

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
1210 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 15, NO. 3, JULY 2018

Fig. 5. Architecture of the complete planning system.

implicitly compute C. But still a major problem is the scaling

of the computational complexity of path planning algorithms
with the structure of the environment map. In case of an
optimal algorithm Aopt , the results scale directly with the
exploration stepsize s. Consequently, it can be observed that
there are two boundary cases, which means for a small
exploration stepsize s↓ the planning results in a principally
shorter path p↓ and a high planning time t p,↑ . The opposite
case is true for a large s↑ , compare the following equations:
Fig. 6. Results task sequencing using different heuristics: results planning
time (left); results task executing time (right). p↓ , t p,↑ ← Aopt (s↓ ) (9)
p↑ , t p,↓ ← Aopt (s↑ ). (10)
The problem is to find a suitable s , for which it is possible
see Fig. 6 (left). Because the NN solution scales in O(N 2 ),
to find a nearly optimal path p↓ ≈ popt in a short planning
with N = |MP| the effect for this small amount of MP
is not visible. For a task sequencing problem, for example time t p,↓ . Also, the problem must be avoided that for s↑
with eight manipulation primitives, the advantage of finding no path will be found. To deal with this problem, we will
the optimal solution in comparison to use the NN solution do develop a method, which uses well-known path planning
not really exist. algorithms and adapt the exploration stepsize automatically
Fig. 5 summarizes the process of task and path planning to the local environment conditions using the knowledge of
in a coherent architecture. Based on the relational model the map occupancy. Therefore, we introduce a preprocessing
created from CAD data, the processor computes the graph of the environment map to compute well-suited stepsizes for
with the symbolic DoFs, which is further used to generate a all subspaces.
1) Adaptive Space Division for Global Path Planning:
set of manipulation primitives. The information is then fused
with the vision data and due to the concept of visibility and Further, we will discuss the theoretical approach to adapt the
exploration stepsize of the global path planning algorithms
accessibility it is possible to get a local set of executable
automatically to a given map. For map representation we use
manipulation primitives. The local set is further sequenced. For
executing the manipulation primitive set also path information a probabilistic voxel map PVM ∈ R3 . Because the map is rep-
resented by a set of voxel iN i = PVM, which are described
is required. So, in Section III-D, we will discuss a new and
more efficient approach for global path planning. by the states {occupi ed o, f r ee f, unknown u} ∈ i
it is possible to introduce a measure of the global occupancy
ρglobal, described by

D. Path Planning j oj
ρglobal =  , with j ≤ i ; ρglobal ∈ [0 . . . 1]. (11)
To compute a collision-free path p = f (x, y, z, ϕ x , i vi
ϕ y , ϕz ) for an entire fixed-base spatial manipulator with The correlation of the exploration stepsize with respect to the
n-revolute joints, the configuration space C is a subset of map occupancy has been chosen as
(R3 × SO(3))n [44], makes it practically impossible to com-
s,↑ ← ρ↓ (12)
pute it completely. To deal with this problem, many search-
or sampling-based path planning algorithms are known, which s,↓ ← ρ↑ . (13)

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
FRIEDRICH et al.: EFFICIENT TASK AND PATH PLANNING FOR MAINTENANCE AUTOMATION USING A ROBOT SYSTEM 1211

With the adaption of s , the planner can explore the path

in free space with a large s,↑ and in more cluttered space
with a smaller exploration stepsize s,↓ . To consider also
the local occupancy, the method divides PVM iterative in
subspaces Ri . This means that PVM represents a tree T and
the corresponding subspaces Ri leafs. In every division, eight
new subspaces are created per parent node in dependence of
the local occupancy of Ri , so that the map can be handled
as an octree. Every node is described by a cuboid C =
Ix × I y × Iz = [x min , x max ]×[ymin , ymax ]×[z min , z max ] which is
divided into eight equal leaf cubes c1 , c2 . . . , c8 . The method
is now straightforward, because the exploration stepsize is
adapted independently of the depth of the tree dT , so
s = g(dT) = g( f (ρ)). (14)
Algorithm 2 and Fig. 7 describes the complete adaptive space
subdivision (ASD) procedure.

Algorithm 2 : Adaptive Space Division (ASD)

Input: Probabilistic Voxel Map PVM.
Output: Probabilistic Voxel Map with space subdivision
PVM AS D and set of stepsizes {s } for corresponding sub-
space Ri
1: // Compute
 global occupancy
j o j ∈PVM
2: ρ ←  ;
i i ∈PVM
3: s  ← g(dT);
4: k = 1; Fig. 7. Schematic 2-D representation of the adaptive space division algorithm.
5: // Start Space Subdivision The building of the tree is done using iterative depth-first.
6: for i = 1:1:8
7: while (ρ > ρlimit ) do second-stage uses oriented bounding boxes for every link. The
8: if (k > 1)  then
kinematic is described through DH-parameters. To smooth the
o j ∈Ri result of the path planner Bernstein-Bezier splines are used.
9: ρ ← j ;
i i ∈Ri 3) Experimental Results (Path Planning): To evaluate the
10: s ← g(dT); results of the adaptive subspace division, we compare it against
11: end if the constant minimal exploration stepsize smin ∈ s and
12: (Ri,1 , Ri,2 , . . . , Ri,8 ) ← divideSpace(Ri ); the constant maximum exploration stepsize smax ∈ s . Two
13: PVM AS D ← {Ri,1 , Ri,2 , . . . , Ri,8 }; scenarios are used to validate the method. Scenario 1 is a
14: s  ← s; typical industrial task such as bin picking and scenario 2
15: k ← k + 1; consists of an occupied space through which the robot must
16: end while move, see Fig. 8 first column. Fig. 8 also summarizes the
17: end for results using an adaptive exploration stepsize in comparison to
18: return (PVM AS D , s ); a constant one. The different exploration stepsizes according
1: function (c1 , c2 . . . , c8 ) ← divideSpace(C) to the subspace occupancy can be seen clearly. It has been
2: c1 ← [x min + 0.5|Ix |, x max ]× found experimentally that a subdivision of dT = 3 is sufficient
[ymin , ymin + 0.5|I y |] × [z min , z min + 0.5|Iz |] and a further subdivision does not offer further computational
3: c2 ← [x min , x min + 0.5|Ix |]× advantages for the applied workspace.
[ymin , ymin + 0.5|I y |] × [z min , z min + 0.5|Iz |] The exploration stepsize is here computed linearly s,dT =
..
4: . 0.5·s,dT −1 with respect to the actual tree depth dT ∈ N+ and
5: c8 ← [x min , x min + 0.5|Ix |]× s,0 = smax . To evaluate, the approach typical metrics [45] are
[ymin + 0.5|I y |, ymax ] × [z min + 0.5|Iz |, z max ] used with:
6: end function 1) path planning time t p in [s];
2) collision detection time tc in [s];
2) Path Planning Algorithms: For path planning we apply 3) path length | p| in [m];
search-based planners with Greedy and A∗ and also sampling- 4) deviation optimal path | p∗ | ∈ {x ∈ R+ |x ≥ 1};
based planners with an RRT and a bidirectional-RRT. The col- 5) path smoothness κ in [rad];
lision detection is done using a two-stage concept. The 6) number of explored vertices |V | ∈ N+ ;
first stage verifies only a sphere to voxel detection and the 7) success rate S ∈ [0 . . . 1] during one minute.

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
1212 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 15, NO. 3, JULY 2018

Fig. 8. Scenarios and results of the ASD approach: results of the ASD algorithm scenario 1 (left first row) and scenario 2 (left second row); exploration with
RRT using scenario 1 (middle first row); exploration with A∗ using scenario 2 (middle second row); resulting path with adaptive, minimum and maximum
exploration stepsize for scenario 1 with RRT (right first row) and scenario 2 with A∗ (right second row).

TABLE II
E VALUATED M ETRICS FOR PATH P LANNING -S CENARIO 1

TABLE III
E VALUATED M ETRICS FOR PATH P LANNING -S CENARIO 2

The experiments are repeated five times for the search-based In almost all cases of search-based planners, the path
planners and thirty times for the sampling-based planners. planning and the collision detection time can be reduced
Table II represents the results for scenario 1 and Table III by up to 70% percent. Also in all cases a path can
for scenario 2, as the arithmetic mean over all experimental be computed in comparison with scenario 2 where it is
runs. Fig. 9 shows the path planning time, the deviation from not possible to find a path using smax and the Greedy
the optimal path and the explored nodes as box-plot. algorithm.

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
FRIEDRICH et al.: EFFICIENT TASK AND PATH PLANNING FOR MAINTENANCE AUTOMATION USING A ROBOT SYSTEM 1213

Fig. 9. Box-Plots of the metric for path planning time: scenario 1 (left first row), scenario 2 (left second row); deviation from optimal path: scenario 1 (middle
first row), scenario 2 (middle second row); number of explored nodes: scenario 1 (right first row), scenario 2 (right second row).

The deviation from the optimal path is also, in most cases,

better when the ASD algorithm is applied than the maximal
exploration stepsize. In some cases, the path of the adaptive
exploration stepsize s results in a larger path length than
using smax , because of the static discretization grid, which is
constant in a subspace Ri . Fig. 10 shows this effect of the
discretization according to the resulting path length.
The ASD is a good compromise for sampling-based plan-
ners, too. The number of explored vertices can be reduced
in all cases and also the improvement of the success rate is
possible through an adapted exploration stepsize.
IV. C ONCEPT FOR I NTERFACING TASKS
AND PATH P LANNING
Fig. 10. Discretization with different exploration stepsizes: Using smin results
In the previous sections the task and path planning have in the minimal path length. With the adaptive exploration stepsize s the path
been described in detail. Obviously, the task sequence is the length is greater than with the maximal exploration stepsize smax because the
discretization grid of smin is closer to the boundaries of the obstacle.
input information for the path planning algorithm. However,
there is a discrepancy in how the information is presented for
the two modules. The task sequencing operates on data in
custom-defined DSL. Using the ANTLR-generated parser, the
a symbolic form, whereas the path planner requires specific
application program can be interpreted on the target system.
Cartesian data.
As a next goal for this ongoing research, the custom parser will
In order to assure a high degree of flexibility of the imple-
be integrated in the robot controller. Based on the application
mentation, also for the future research, the serialization of the
program, the parser will call the skill primitives [31] of the
manipulation primitives (output of task sequencing) has been
robot with the correct parameterization. Thus, the parser will
carried out using a custom domain-specific language (DSL).
link the symbolic manipulation primitives with skill primitives,
This is defined as close to the theoretical description of manip-
which are a proven paradigm for the execution of complex
ulation primitives as possible. The interpreter for this DSL has
robot tasks. Further advantages of using a DSL are:
been automatically generated based on the extended Backus-
Naur form of the language syntax using the ANother Tool 1) close representation of the mathematical formalism of
for Language Recognition (ANTLR) parser generator. ANTLR general task planning;
can generate a parser based on the syntax definitions in a wide 2) flexible serialization of commands and even logic if
range of languages (e.g., C#, Java, and C++), which can be needed; and
integrated in the target system (e.g., as an ROS node). The 3) human-readable application program (similar for all con-
task sequencer generates an application program written in this trol equipment in a factory setting).

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
1214 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 15, NO. 3, JULY 2018

V. C ONCLUSION [18] A. P. Ambler and R. J. Popplestone, “Inferring the positions of bod-

ies from specified spatial relationships,” Artif. Intell., vol. 6, no. 2,
In this paper, an approach for task and path plan- pp. 157–174, 1975.
ning was investigated, which allows a robot system to [19] L. S. Homem de Mello and A. C. Sanderson, “AND/OR graph repre-
sentation of assembly plans,” IEEE Trans. Robot. Autom., vol. 6, no. 2,
autonomously plan different maintenance tasks. Using a novel pp. 188–199, Apr. 1990.
sampling-based approach to compute the relative DoF between [20] T. D. Fazio and D. Whitney, “Simplified generation of all mechani-
components in combination with the symbolic concept of cal assembly sequences,” IEEE J. Robot. Autom., vol. RA-3, no. 6,
pp. 640–658, Dec. 1987.
manipulation primitives, a powerful method for task planning
[21] K. E. Moore, A. Gungor, and S. M. Gupta, “Disassembly process
is available. Further, a universal concept for reducing the planning using Petri nets,” in Proc. IEEE Int. Symp. Electron. Environ.,
planning time of global path planners is provided. An actual May 1998, pp. 88–93.
limitation of the task planning approach is that it works only [22] L. S. Homem de Mello and A. C. Sanderson, “Representations of
mechanical assembly sequences,” IEEE Trans. Robot. Autom., vol. 7,
for a single-arm manipulator. In the future it will be extended no. 2, pp. 211–227, Apr. 1991.
so that it works also for dual-arm manipulation. Also, we [23] A. Gungor and S. M. Gupta, “Disassembly sequence planning for
will focus on the development of the parser, which enables complete disassembly in product recovery,” in Proc. Northeast Decision
Sci. Inst. Conf., 1998, pp. 25–27.
interfacing planning and execution. [24] U. Thomas, “Automatisierte Programmierung von Robotern für
Montageaufgaben,” Ph.D. dissertation, Inst. Robotik Prozessinformatik,
Tech. Univ. Braunschweig, Aachen, Germany, 2008.
R EFERENCES [25] R. H. Wilson and J. C. Latombe, “Geometric reasoning about mechanical
[1] E. Abele, M. Dervisopoulos, and B. Kuhrke, “Bedeutung und Anwen- assembly,” Artif. Intell., vol. 71, no. 2, pp. 1–31, 1992.
dung von Lebenszyklusanalysen bei Werkzeugmaschinen,” in Leben- [26] L. S. Homem de Mello and A. C. Sanderson, “A correct and complete
szykluskosten Optimieren: Paradigmenwechsel für Anbieter und Nutzer algorithm for the generation of mechanical assembly sequences,” IEEE
von Investitionsgütern, S. Schweiger, Ed. Wiesbaden, Germany: Gabler, Trans. Robot. Autom., vol. 7, no. 2, pp. 228–240, Apr. 1991.
2009, pp. 51–80. [27] H. Mosemann and F. M. Wahl, “Automatic decomposition of planned
[2] R. Isermann, Fault-Diagnosis Systems: An Introduction From Fault assembly sequences into skill primitives,” IEEE Trans. Robot. Autom.,
Detection to Fault Tolerance. Berlin, Germany: Springer-Verlag, 2006. vol. 17, no. 5, pp. 709–718, Oct. 2001.
[3] R. Isermann, Fault-Diagnosis Applications: Model-Based Condition [28] F. Röhrdanz, H. Mosemann, and F. M. Wahl, “HighLAP: A high level
Monitoring: Actuators, Drives, Machinery, Plants, Sensors, and Fault- system for generating, representing, and evaluating assembly sequences,”
Tolerant Systems. Berlin, Germany: Springer-Verlag, 2011. in Proc. IEEE Int. Joint Symp. Intell. Syst., 1996, pp. 134–141.
[4] K. Toussaint, N. Pouliot, and S. Montambault, “Transmission line [29] U. Thomas, M. Barrenscheen, and F. M. Wahl, “Efficient assembly
maintenance robots capable of crossing obstacles: State-of-the-art review sequence planning using stereographical projections of C-space obsta-
and challenges ahead,” J. Field Robot., vol. 26, no. 5, pp. 477–499, 2009. cles,” in Proc. IEEE Int. Symp. Assembly Task Planning, Jul. 2003,
[5] P. Debenest et al., “Expliner—Robot for inspection of transmis- pp. 96–102.
sion lines,” in Proc. IEEE Int. Conf. Robot. Autom., May 2008, [30] U. Thomas and F. M. Wahl, “Assembly planning and task planning—
pp. 3978–3984. Two prerequisites for automated robot programming,” in Robotics Sys-
[6] C. Birkenhofer, K. Regenstein, J. M. Zöllner, and R. Dillmann, “Archi- tems for Handling and Assembly, B. Siciliano, O. Khatib, and F. Groen,
tecture of multi-segmented inspection robot KAIRO-II,” in Robot Motion Eds. Berlin, Germany: Springer-Verlag, 2010, pp. 333–354.
Control. London, U.K.: Springer, 2007, pp. 381–389. [31] U. Thomas, B. Finkemeyer, T. Kröger, and F. M. Wahl, “Error-tolerant
[7] L. Pfotzer, S. Ruehl, G. Heppner, A. Roennau, and R. Dillmann, execution of complex robot tasks based on skill primitives,” in Proc.
“KAIRO 3: A modular reconfigurable robot for search and rescue field IEEE Int. Conf. Robot. Autom., Sep. 2003, pp. 3069–3075.
missions,” in Proc. IEEE Int. Conf. Robot. Biometrics, Dec. 2014, [32] F. Torres, S. Puente, and C. Diaz, “Automatic cooperative disassembly
pp. 205–210. robotic system: Task planner to distribute tasks among robots,” Control
[8] X. Gao et al., “Multifunctional robot to maintain boiler water-cooling Eng. Pract., vol. 17, no. 1, pp. 112–121, 2009.
tubes,” Robotica, vol. 27, no. 6, pp. 941–948, 2009. [33] S. Vongbunyong, S. Kara, and M. Pagnucco, “Basic behaviour control
[9] G. Paul, S. Webb, D. Liu, and G. Dissanayake, “Autonomous robot of the vision-based cognitive robotic disassembly automation,” Assembly
manipulator-based exploration and mapping system for bridge mainte- Autom., vol. 33, no. 1, pp. 38–56, 2013.
nance,” Robot. Autonomous Syst., vol. 59, nos. 7–8, pp. 543–554, 2011. [34] S. Vongbunyong, S. Kara, and M. Pagnucco, “Application of cognitive
[10] G. Paul, P. Quin, C.-H. Yang, and D. Liu, “Key feature-based approach robotics in disassembly of products,” in CIRP Ann., vol. 62, no. 1,
for efficient exploration of structured environments,” in Proc. IEEE Int. pp. 31–34, 2013.
Conf. Robot. Biomimetics, Dec. 2015, pp. 90–95. [35] S. M. LaValle, Planning Algorithms. Cambridge, U.K.:
[11] B. Kuhlenkotter, M. Buecker, and T. Brutscheck, “Deploying mobile Cambridge Univ. Press, 2006.
maintenance robots in material flow systems using topological maps and [36] L. E. Kavraki and S. M. LaValle, “Motion planning,” in Hand-
local calibration,” in Proc. 6th German Conf. Robot. Int. Symp. (VDE), book of Robotics, B. Siciliano and O. Khatib, Eds. Berlin, Germany:
Jun. 2010, pp. 1–7. Springer-Verlag, 2016, pp. 139–161.
[12] M. Bengel, K. Pfeiffer, B. Graf, A. Bubeck, and A. Verl, “Mobile robots [37] P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic
for offshore inspection and manipulation,” in Proc. IEEE/RSJ Int. Conf. determination of minimum cost paths,” IEEE Trans. Syst. Sci. Cybern.,
Intell. Robots Syst., Oct. 2009, pp. 3317–3322. vol. SSC-4, no. 2, pp. 100–107, Jul. 1968.
[13] K. Pfeiffer, M. Bengel, and A. Bubeck, “Offshore robotics—Survey, [38] M. Likhachev, G. Gordon, and S. Thrun, “ARA∗: Anytime a∗ with
implementation, outlook,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots provable bounds on sub-optimality,” in Advances in Neural Information
Syst., Sep. 2011, pp. 241–246. Processing Systems. Cambridge, MA, USA: MIT Press, 2004.
[14] P. Jiménez, “Survey on assembly sequencing: A combinatorial and [39] S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach.
geometrical perspective,” J. Intell. Manuf., vol. 24, no. 2, pp. 235–250, Englewood Cliffs, NJ, USA: Prentice-Hall, 2009.
2013. [40] S. M. LaValle and J. J. Kuffner, “Rapidly-exploring random trees:
[15] S. Ghandi and E. Masehian, “Review and taxonomies of assembly and Progress and prospects,” in Proc. Workshop Algorithmic Found. Robot.,
disassembly path planning problems and approaches,” Comput.-Aided 2000.
Design, vols. 67–68, pp. 58–86, Oct. 2015. [41] S. Karaman, M. R. Walter, A. Perez, E. Frazzoli, and S. Teller, “Anytime
[16] A. J. D. Lambert, S. M. Gupta, Disassembly Modeling for Assembly, motion planning using the RRT*,” in Proc. Int. Conf. Robot. Autom.,
Maintenance, Reuse and Recycling. Boca Raton, FL, USA: CRC Press, May 2011, pp. 1478–1483.
2005. [42] C. Friedrich, A. Lechler, and A. Verl, “A planning system for generating
[17] H. Mosemann, “Beiträge zur Planung, Dekomposition und Ausführung manipulation sequences for the automation of maintenance tasks,” in
von automatisch generierten Roboteraufgaben,” Ph.D. dissertation, Proc. IEEE Int. Conf. Autom. Sci. Eng., Aug. 2016, pp. 843–848.
Inst. Robotik Prozessinformatik, Tech. Univ. Braunschweig, Aachen, [43] G. Marsaglia, “Choosing a point from the surface of a sphere,” Ann.
Germany, 2000. Math. Statist., vol. 43, no. 2, pp. 645–646, Apr. 1972.

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.
FRIEDRICH et al.: EFFICIENT TASK AND PATH PLANNING FOR MAINTENANCE AUTOMATION USING A ROBOT SYSTEM 1215

[44] B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo, Robotics: Mod- Armin Lechler was born in Bietigheim-Bissingen,
elling, Planning and Control. London, U.K.: Springer-Verlag, 2009. Germany, in 1979. He received the Dipl.-Ing. degree
[45] B. Cohen, I. A. Şucan, and S. Chitta, “A generic infrastructure for in technical cybernetics with emphasis on produc-
benchmarking motion planners,” in Proc. IEEE/RSJ Int. Conf. Intell. tion engineering from the University of Stuttgart,
Robots Syst. (IROS), Oct. 2012, pp. 589–595. Stuttgart, Germany, in 2006.
He was a Research Assistant at the Institute for
Control Engineering of Machine Tools and Manufac-
Christian Friedrich was born in Heilbronn- turing Units (ISW), University of Stuttgart, in 2009,
Neckargartach, Germany, in 1986. He received the where he was involved in research in information
B.Eng. degree in robotics and automation and the and communication technology. From 2009 to 2011,
M.Eng. degree in electronic systems engineering he was the Head of the Department Control Engi-
from the University of Applied Science Heilbronn, neering, ISW. Since 2009, he has been the Managing Director of FISW
Heilbronn, Germany, in 2013. Steuerungstechnik GmbH, Stuttgart. In 2011, he joined ISW as the Managing
Since 2013, he has been with the Institute for Chief Engineer, completed his dissertation in Dr.-Ing. Since 2013, he has been
Control Engineering of Machine Tools and Man- the Deputy Director of ISW.
ufacturing Units, Graduate School of Excellence
Advanced Manufacturing Engineering, University Alexander Verl was born in Frunse, Russia, in
Stuttgart, Stuttgart, Germany. His current research 1966. He received the Dipl.-Ing. degree in electri-
interests include planning and control in robotics. cal engineering from Friedrich-Alexander-University
Erlangen-Nürnberg, Erlangen, Germany, in 1991,
and the Dr.-Ing. degree in control engineering from
the DLR Institute of Robotics and Mechatronics,
Oberpfaffenhofen, Germany, in 1997.
Akos Csiszar was born in Satu Mare, Roma- From 1997 to 2005, he was Founder and Managing
nia, in 1985. He received the Diploma degree in Director of AMATEC Robotics GmbH (part of
mechatronics from the Technical University of Cluj- KUKA Roboter GmbH since 2005), Augsburg,
Napoca, Cluj-Napoca, Romania, in 2010. In 2010, Germany. Since 2005, he has been a Full Professor
he became a Ph.D. Student with the Technical Uni- with the University of Stuttgart, Stuttgart, Germany, where he was the Head
versity of Cluj-Napoca and in 2011, having granted a of the Institute for Control Engineering of Machine Tools and Manufacturing
DAAD Scholarship, he became a Ph.D. Student with Units.
the University of Stuttgart. He finished his Ph.D. Prof. Verl received the Diesel Gold Medal from the German Institute of
research in the field of path planning for industrial Inventions in 2014, the Julius von Haast Fellowship Award from the Royal
robots with a joint Ph.D. title from both universities. Society of New Zealand in 2012, the Honorary Doctoral degree (Dr.h.c.)
Since 2013, he has been a junior research group of the Technical University of Cluj-Napoca, Romania, in 2012, the Honorary
leader in intelligent production systems with the Graduate School of Excel- Professor of The University of Auckland, New Zealand, in 2012, the Invention
lence advanced Manufacturing Engineering, University of Stuttgart, Stuttgart, and Entrepreneurship Award of the IEEE Robotics and Automation Society,
Germany. His current research interests are applications of optimization the International Federation of Robotics in 2010, and the Dr.h.c. of the
methods for improving production systems. “Politehnica” University of Timisoara, Romania, in 2009.

Authorized licensed use limited to: Jonkoping University. Downloaded on October 28,2022 at 10:43:52 UTC from IEEE Xplore. Restrictions apply.