5.3.

Iterative Matrix Solver and Convergence Analysis for Linear Random Flow Problems
5.3.1. Review of iterative and preconditioned matrix solvers
(from Ababou et al. 1988: pp. 343-360)

linearization, and Fourier space representations,

REFERENCES: could be used

Ababou (1988): Three-Dimensional

to R.analyze mesh resolution Flow in Random Porous Media.for
requirements Ph.D. more
thesis. Ralph Parsons Lab. for
general
Water Resources & Hydrodynamics, Department of Civil Engineering, Massachusetts Institute of Technology,
Cambridge, Massachusetts
stochastic 02139, USA.
partial (3 vols., 833 pp.,equations,
differential March 1988). such as parabolic

Available from MIT at: http://libraries.mit.edu/

convection-diffusion equations.http://dspace.mit.edu/handle/1721.1/14675
The present work could also be
Also published as MIT Technical Report N°318:
used to obtain a more complete picture of the discretization
Ababou R., L.W.Gelhar, D.McLaughlin (1988): Three-Dimensional Flow in Random Porous Media. MIT
Technical
errors,Report N° 318.
such as Ralph
the Parsons
spatial Laboratory for Water Resources
correlation structure and of
Hydrodynamics,
the errors Department of
Civil Engineering, Massachusetts Institute of Technology (MIT), Cambridge, Massachusetts 02139, USA. (833 pp.,
March
on1988).
the hydraulic head and on the flux vector. Other possible

------------------------------------------------------------------------------------------------------------------------------------------
applications could involve the study of higher order finite

This document reproduces

difference subsection
methods. 5.3.1. "Review
finite of iterative
element and preconditioned
methods. matrix solvers",
pseudo-spectral
pp. 343-360 from Ababou (1988) and/or Ababou, Gelhar, McLaughlin (1988).
methods, and multigrid methods for stochastic PDE's with random

coefficients.

5.3 Iterative Matrix Solver and COnvergence Analysis

for Linear Random Flow Problems

5.3.1 Reviewof iterative and preconditioned matrix solvers

Our surveyof the literature (Table 5.3) clearly showed

that large sparse matrix systems can be solved more efficiently

with iterative solvers than with direct solvers such as Choleski

factorization or Gauss substitution. One of the Many examples of

the superiority of iterative solvers. even for relatively modest

 Table 5.3 List of Natrix Solvers and References

Solvers Comments Applications to Subsurface

Hydrology and Related Problems

Point Jacobi Explicit type; O{n 2 ) seldom used

iterations.

2. Gauss Seidel Weakly implicit; O(n 2 ) seldom used

iterations.

3. SOR Accelerated Gauss-Seidel Reisenauer et al .. 1981

D(n) iterations with Bjordammen and Coats. 1969
optimal parameter. else Stone 1968
D(n 2 ) iterations.

4. LSOR Li ne - SOR. i mp 1ici t Bjordammen and Coats. 1969

along lines; requires Freeze. 1971. Coo ley. 1974
optimal parame ter for
D(n) convergence.
5 ADI Implicit along each Stone 1968
direction alternatively: Weinstein et al .. 1969
requires near-optimal Bjordammen and Coats. 1969
sequence of parameters. Watts 1971: Cooley. 1974:
Trescott and Larson 1977:
Kershaw. 1978

6. Iax; First order accurate Kershaw 1978

Incomplete Choleski Gambolati. 1979
factorization. with Kuiper 1981 and 1987
Conjugate Gradient Gambolati and Perdon 1984
iterations (no parameter
required) .

7. SIP Second order accurate Stone 1968. Weinstein et. al.

strongly implicit LU 1969. Cooley 1974 and 1983.
factorization (requiring Trescott. 1975. Trescott and
sequence of parameters); Larson 1977. McDonald and
Picard iterations. Harbaugh. 1984. Kuiper 1981
and 1987.
- --,
8. Gauss Fully implicit. direct Neuman and fuvis 1983
El i mi na t i on solver (non-iterative). Yeh and Luxmoore 1983
or Choleski
Factorization
 3~5

size problems, can be found in Gambolati and Perdon (1984). Our

own experience wi th a Gauss elimination solver adapted to a

banded Galerkin coefficient matrix confirmed this view. The CPU

7/3
time for this direct solver was proportional to N for steady

state 3D groundwater flow on a cubic domain discretized into N

elements. On a Vax 111782 machine, the CPU time was about one

hour for N = 4000 elements. However, solving a problem of size

N = 64000 would have required one month of CPU time on the same

macbine, and larger problems on the order of 1 million elements

could not be solved in reasonable amounts of time even on recent

supercomputers such as Cray 2. The storage would be likewise

prohibi tive. being proportional to N5 / 3 (number of equations N

times the matrix bandwidth N2/3 ). Thus, the storage requirement

will be 10 Gigawords for large problems on the order of 1 million

elements. In comparison, the central memory of the Cray 2 is

currently about 250 Megawords.

The major disadvantage of using direct solvers for the

solution of large linear systems lies in the fact that the

triangular matrices arising in the process of decomposition are

not sparse, even though the coeff icient matrix i tself may be

sparse. For the 7-diagonal finite difference matrix depicted in

Figure 5.2, a Choleski factorization A = LLT yields a triangular

matrix L with mostly non-zero elements within the half band of

width n 2 = N2 / 3 (for a cubic grid of size As

 346

mentioned above, this yields on the order of non-zero

elements to be computed, compared to just 4N non-zero elements in

the lower balf of the original matrix. This kind of observation

bas led numerical analysts to develop a number of i terative

solution methods based on approximate sJXLrse decomposi tions of

the original system matrix, such that the computational work per

iteration and the required storage are both proportional to N.

The sparse iterative methods can be roughly classified

with respect to the approximate decomposition method used. Most

of the "classical" iterative solvers are based on an approximate

spU t ting of the matrix (Point Jacobi. Gauss-Seidel. successive

overrelaxation (SOR). alternate directions implicit (ADI» while

the more recent "fast" iterative solvers are based on an

approximate Factorization of the matrix (strongly implicit

procedure, and incomplete Choleski-conjugate gradients). The

reader is referred to Jacobs (1981) for a survey of iterative

solvers according to the classification proposed above, and Evans

(1981) for a review of matrix-spli tting preconditioners. A

number of other reviews and experimentations wi th matrix

i terative methods can be found in the collections of papers

edited by Schultz (1981), Evans (1983), and Birkhoff and

Schoenstadt (1984). In addi tion, Table 5.3 gives a list of

references concerning the use of iterative solvers for subsurface

flow and analogous problems; sorne of these studies include

 347

numerical experiments and comparisons between different types of

solvers.

Another important distinction to be made among

iterative solvers is. precisely. the type of iteration used to

converge to the solution of the original matrix system. Apart

From the lOOG solver. aIl the other solvers mentioned above are

based on Picard-like iterations. including in particular the SIP

methoo.. These solvers can be briefly described as follows.

Consider the linear finite difference system:

Ah =b (5.80)

and suppose that an approximation M of matrix A has been

found (M mus t be easier to invert than A) . The simple

manipulation shown below leads quite naturally to a Picard

iteration scheme where the new system matrix M is by

construction easier to invert than the original matrix A:

(M + A - M ) h = b

Mh =b + (M-A)h

Mhm+l =b + (M-A)h m.
 348

Furthermore, by substracting Mh m to both sides. one obtains a

"modified Picard" scheme that is presumably more stable wi th

respect to round-off errors:

M 0 ohm+ 1 =b - Ah m

where m is the iteration counter.

Finally. the i terations can be overrelaxed or underrelaxed by

mul tiplying the right-hand side residual by a "relaxat ion

parame ter" w:

IM Ohm+1 = wo(b-Ah m)
O (5.81)
1_ _ _ _ __

It is worth noting that (5.81) is a consistent

i teration scheme with respect to the original system (5.80), in

the sense that the exact solution h = A-lb is obtained as m ~ m.

provided however that the i terations converge. Unfortunately,

convergence is not necessarily guaranteed in the general case.

We now proceed to review various kinds of

preconditioners (matrix M). The classical iterative solvers such

as Jacobi and various versions of successive overrelaxation (SOR)

are based on an approximate decomposi tion obtained by spI i tting

A into lower triangular. diagonal. and upper triangulaI'

 349

matrices:

A=L + D+ U

The approximate matrix M can be expressed in the general form:

(5.82)

When the preconditioner (5.82) is plugged into (5.81), one

obtains sorne weIl known i terative solvers of the

"matrix-spli t ting" kind:

Point Jacobi (1'L = l'u = 0, and w = 1)

Point Gauss-Seidel (1'L = l, l'
u = 0, andw = l)
Point SOR (1'L = l, l'
u = O. and 1 ~ Iwl~ 2)

Symmetric SOR (1'L = l' U = 1'. and w = 1'(2-1')}

Similarly. the ADI solver can be viewed as an iterative

method based on matrix-splitting decomposition. As an example.

the Peaceman-Rachford version of ADI for two-dimensional finite

difference systems (Peaceman and Rachford, 1955). can be

expressed as follows:
 350

where the and Ly matrices correspond to the partial

differential operators in the X and Y-directions, respectively.

Note that each step of AD! is similar to the basic i terative

scheme (5.81). The AD! method can be extended to

three-dimensional finite difference systems (Douglas, 1962).

More general alternate directions operator splitting methods have

also been devised for the solution of weighted residual and

collocation systems (Celia and Pinder, 1985).

The convergence properties of sorne of the classical

iterative solvers reviewed above have been thoroughly analyzed in

the literature (Varga 1962, Young 1971, Golub and Van Loan 1983).

For instance, i t has been shown that Jacobi and Gauss-Seidel

require on the order of n2 iterations to reach a given

precision, where n is the unidirectional size of the grid

(Laplace problem, in any number of dimensions). The SOR methods

require only O(n) iterations if the optimal iteration parameter

can be computed accurately. Unfortunately, this requires

estimating the spectral radius of the SOR i teration matrix. For

 351

complex problems such as the stochastic flow equation, it seems

unI ikely that the "optimal" relaxation parame ter could be

es timated accurately wi thou t dramatically increasing the total

computational work. Thus. the number of iterations is likely to

be O(n 2 ) - rather than the O(n) behaviour predicted for optimat

SOR.

This feature is presumably shared by the iterative AD!

methods. The theory for AD! convergences remains incomplete. but

the work of Peaceman and Rachford (1955) and Wachspress and

Habeter (1960) shows that the optimal ADI-relaxation parameter is

not a constant. These authors proposed a cycl ic sequence of

parameters based on the eigenspectrum of the Lx and

matrices defined above. The truly optimal sequence is not lmown.

except for very special forms of the governing equation. e.g ..

the heat equation with spatially separable conductivi ty

K(x.y)=K (x)-K (y). More detai ls can be found in Varga (1962)

x y
and Ames (1977). among others.

The Jacobi. SOR. LSOR, and ADI solvers can also be

compared in a slightly different way as follows. First of aIl,

let us point out key feature shared by aIl the iterative methods

reviewed above: the interactions among nodal variables are

part iaUy decoupLed through the iterative solution process. At

each i teration step. the solution is computed by taking into

 352

account the interactions among a certain group of nodal values,

while the remaining nodes are treated explicitly (e.g., by

retaining the values obtained at the previous i teration). The

cornputational work per i teration is roughly the same for aIl

solvers (on the order of N), but the "degree of irnplici tness"

differs. In the Jacobi method, the solution at each node is

cornputed explicitly with respect to aIl the remaining nodes: in

the Gauss-Seidel and SOR methods, about half of the nodes are

treated irnplicitly on average: and in the ADI method, the nodal

values are coupled al ternatively in the X and Y-directions,

while the other direction is treated explicitly. Various devices

have been proposed in the 1 i terature in order to increase the

degree of implicitness, or coupling, of iterative solvers while

still retaining the advantages of a sparse and easily invertible

rnatrix approximation. Indeed, the SOR matrix-splitting can be

generalized into line-SOR or more generally block-SOR splittings,

which rnay increase the coupling along lines or among neighboring

nodes (Evans, 1984).

However, our literature review indicates that the most

efficient, or "strongly irnpl ici t", i terative solvers are those

based on an approximate factorization of the original rnatrix,

such as the SIP and Ia:lj solvers mentioned earlier. This was

taken into account in the classification given in Table 5.3,

where the solvers were listed according to their "degree of

 353

impl ici tness" , increasing from top to bottom (Jacobi, Gauss

Seidel. SOR. LSOR. ADI. IOOG. SIP. and Direct Solvers). According

to this classification. direct solvers are fully implicit

(requiring only "one i teration") but their computational cost

will be prohibitive for large systems, as explained earlier. The

advantage of SIP and lOOG is that they are based on strongLy

impL ici t, yet sparse factorizations of the coefficient matrix.

Thus, these two methods presumably converge faster than the

solvêrs based on matrix-spli tting, while the computational work

per iteration remains on the order of N.

The numerical experiments publ ished in the 1 i terature

confirmed this view. Most of the references given in Table 5.3

above involved comparisons between LSOR, ADI, SIP, and lOOG. It

appeared that ADI had the slowest convergence rate in most cases,

or diverged in difficult cases such a those involving anisotropie

conductivities. The LSOR solvers were reasonably efficient,

provided alternate line-sweeping along different directions, but

the SIP solver was usually more efficient for "difficul t"

problems involving heterogenei ties and mild nonl ineari ty. In

addition, the performance of SIP was not overly sensitive to the

choice of its iteration parameters, whereas this was sometimes a

critical issue for LSOR and ADI.

 354

On the other hand, lOOG also appeared to be a powerful

solver. The numerical experiments by Kershaw (1978) demonstrate

the superiori ty of lOOG over ADI and LSOR for a laser fusion

problem (transient diffusion equation wi th a radiation term).

Other numerical experiments for heterogeneous confined and

unconfined groundwater flow (Kuiper 1981) indicate that SIP with

underrelaxation could be as efficient as Iax: for the mi ldly

nonlinear case of unconfined flow, although IOCG usually

converged faster for linear problems. In a more recent study

Kuiper (1987) concludes in favor of IOCG over SIP. However, it

is possible that a change in the details of implementation,

especially for the nonlinear problems, could affect his

conclusions. Furthermore, the SIP sol ver involves an adjustable

sequence of iteration parameters (similar to ADI), and could also

be underrelaxed to avoid divergence in difficult cases. On the

other hand, IOCG does not depend on any extraneous i terat ion

parameter. It is conceivable that the flexibility of SIP could

be an advantage, rather than a drawback, when dealing with near

il1-conditioned systems.

At any rate, it may be preposterous to draw definite

conclusions here, since the numerical experiments mentioned above

were limi ted to rather modest-size flow problems, below 10,000

nodes. The largest among those was the saturated-unsaturated

numerical simulation by Freeze (1971) with the LSOR solver on an

 355

8,OOO-node grid, but his study did not include comparisons with

other solvers. Larger simulations can be founà in the literature

on numerical analysis, however, these focus typically on the

solution of the Laplace or Poisson equations (constant

coefficients). One of these studies, by Jacobs (1983), develops

an adaptation of the SIP factorization in conjunction with

conjugate gradient iterations, for comparison with ICOG methods.

The conclusions, based on two-dimensional tes t problems up to

40,000 nodes, were in favor of ICOG over SIP-CG. However, wc do

not lmow of any numerical experiments wi th the standard SIP

solver for problems of comparable size or larger.

We now focus our review on the theory of SIP and IOOG,

since our search in the literature indicates that these solution

methods may have the best potential for large finite difference

systems. The idea of using sparse approximate factorization for

precondi tioned i terative solvers arose in 1968, when the SIAM

JournaL on NumericaL AnaLysis published in the same issue three

papers on the approximate factorization and iterative solution of

multi-dimensional finite difference systems. The f i r st, by

Stone, described the strongLy impLicit procedure (SIP) based on

an approximate, non-symmetric LU factorization of the symmetric

coefficient matrix A, with a Picard iteration scheme to converge

to the solution of the A-matrix system. Although Stone's paper

(1968) concerned only two-dimensional 5-point fini te difference

 356

systems, the SIP method was subsequently extended to

three-dimensional 7-point finite difference systems (Weinstein et

al, 1969). The two other papers in the 1968 SIAM JournaL, by

DuPont, Kendall, Rachford and the companion paper by Dupont,

developed a symmetric LLT approximate factorization with Picard

iterations to converge to the solution. The LLT factorization

they used was similar to an incomplete Choleski factorization

where the matrix L is forced to have the same sparsity pattern

as A, and furtherrnore the row-surns of A are conserved (see

Gustaffson 1978, Jackson and Robinson 1985). Thus. the iterative

solvers of Stone (1968) and Dupont et al. (1968) differed

essentially in the rnethod used to obtain an approximate

factorization of the finite differencc matrix.

It was not unti 1 1977 that the symmetric incomplete

Choleski factorization was used as a preconditioner for conjugate

gradient iterations (Mejerink and Van der Vorst. 1977). This

cornbination, known as incornplete Choleski-conjugate gradients

(ICOG). has bec orne quite popular due to the fast convergence of

the i terations in the case of well conditioned

(preconditioned) symmetric positive-definite systems (see

Kershaw. 1978, among others). It is interesting to note that the

"pure" conjugate gradients method devised by Hestenes and Stiefel

(1952) was viewed in the early days as an exact solver. since the

method was known to converge to the exact solution in at most N

 357

iterations (N being the number of equations). However, the CG

i teration did not converge fast enough to be competi tive as an

approximate large sparse matrix solver (N large). Thus, it

should be kept in mind that the incomplete factorization is an

essential ingredient of the IOOG method, required to ensure fast

convergence of the conjugate gradient iterations.

On the other hand, it is worth noting that the

conjugate gradients method, and the IOOG solver, can only be used

to solve symmetric positive-definite systems. As a consequence,

the conjugate gradients method cannot be used to accelerate the

convergence of the SIP solver, as the latter is based on a

non-symmetric LU factorization. This would seem to advantage the

IOOG method, since the conjugate gradients iterations presumably

converge faster than Picard iterations for weIl conditioned

(preconditioned) systems. On the other hand, the non-symmetric

SIP factorization appears to be a more accurate approximation of

the original system matrix than the Incomplete Choleski

factorization (respectively second order and first order in Ax:

see Stone 1968 and Gustaffson 1978). This seems to advantage

SIP, with a better preconditioner than IOOG.

Unfortunately, there does not appear to be any solid

theoretical basis on which to compare the two methods. A formaI

theory of SIP convergence is still lacking, due to the complex

 358

form of the LU factorization involved: there has not been much

progress in this area since the indications given by Stone (1968)

for 2-dimensional problems with constant coefficients. The

theory for lCO:; is more developed, but limited to constant or

mi Idly variable coefficients. For instance, Gustaffson (1978)

showed that the condition number of the iteration matrix for the

Picard- Incomplete Choleski solver of Dupont, et al. (1968) was

D(n), compared to O(n 2 ) for the condition number of the original

matrix (Laplace equation, with n the unidirectional size of the

grid). On the other hand, the number of iterations required to

reach a given precision grows like the square-root of this

condition number, both for the Picard and the conjugate gradient

iterations (Gustaffson 1978, and Golub and Van Loan 1983). This

yields for the number of rco:; iterations a relation of the form:

~
m n

which indicates that the lCO:; method could converge qui te fast

for large 3D systems. Indeed, the total size of the grid is

N = n3 in three dimensions, which yields:

(5.83)

indicating a very slow growth of the number of i terations wi th

grid size. Note however that the theoretical analysis that led
 359

to (5.83) was based on a number of assumptions, including the

restriction to mildIy variable or constant coefficients. It

seems more reasonable to postulate that, in the worst case, the

number of iterations couid grow Iike the unidirectionai size of

the grid, for the ICOG as weIl as the SIP methods, i.e.:

m n

This gives finally a worst case estimate of the number of

i terations required for convergence of both ICOG and SIP for

lhree-dimensional systems with highly variable coefficients:

(5.84)

In comparison. note that the non-optimal SOR method will not

converge faster than N2 / 3 i terations. even for mildly variable

coefficients.

We have developed a Fortran implern~ntation of the SIP

solver during the initial stages of this research. Sorne of the

detai ls of this implernentation wi Il be descr-ibed in the next

section. and a number of numerical experirnents for large randorn

flow problems will also be presented in a forthcorning section.

Because the results obtained with SIP were eventually found to be

qui te satisfactory. i t was fel ~ that developing the lCOG solver

 360

was not necessary. However, there is no claim that lOOG could not

perform as weIl or perbaps better than SIP in terms of

computational work (more on this later).

5.3.2 Formulation of the strongly implicit procedure (SIP solver)

We now proceed to describe the algebraic details of the

s trongly impl ici t procedure. Recall that SIP is based on the

Picard iteration scheme (5.SI):

[(5.S1)]

where M = LU is an approximate non-synunetric factorization of

the system matrix A. In what follows, we analyse in sorne detail

the SIP factorization for the case of the 7-diagonal symmetric

coefficient matrix A, corresponding to the 7-point centered

finite difference scheme in three dimensions (see Figures 5.1 and

5.2 above). The 3D version of SIP was exposed briefly by

Weinstein et al. (1969) , based on the 2D version previously

developed by Stone (l96S). Details on coding can be found for

instance in McDonald and Harbaugh (19S4). However, our

particular implementation is exposed below.

The SIP factorization aims at obtaining a close

approximation of matrix A in the form of a product of a lower and