JOURNAL OF COMPUTATIONAL PHYSICS 91, 1lo-131 (1990)

Neural Algorithm for Solving Differential Equations

HYUK LEE

Department of Electrical Engineering, Polytechnic Institute of New York,

Brooklyn, New York 11201

AND

IN SEOK KANG

Department of Chemical Engineering, California Institute of Technology,

Pasadena, California 91125
Received August 17, 1988; revised October 6, 1989

Finite difference equations are considered to solve differential equations numerically by

utilizing minimization algorithms. Neural minimization algorithms for solving the tinite dif-
ference equations are presented. Results of numerical simulation are described to demonstrate
the method. Methods of implementing the algorithms are discussed. General features of the
neural algorithms are discussed. 0 1990 Academic Press, Inc.

I. INTRODUCTION

Numerical computation in many disciplines, such as physics, applied mathe-

matics, electrical engineering, biochemistry, etc., has received a great deal of atten-
tion recently as a practical technique to understand complex phenomena that are
almost impossible to treat analytically [ 11. Supercomputers have been built to
speed up the calculation. Furthermore, new computing algorithms based on the
concept of concurrent processing have been developed and implemented by con-
necting a small number of processors.
Recently, highly parallel neural networks have been investigated extensively to
solve complicated problems such as pattern recognition and combinatorial
optimization [2]. Linear simultaneous equations also have been treated by
applying neural networks [3]. Implementation of neural networks by utilizing
volume holographic optical interconnections have proved to be promising [4].
One of the most general methods of solving differential equations is to use finite
difference equations and to solve the algebraic equations [S]. The computational
load for solving the difference equations increases very fast as the number of dis-
crete points becomes large. Therefore, a highly parallel algorithm to solve the finite
difference equations is essential when a complicated problem is encountered. In this
paper, neural algorithms for minimization are utilized to develop highly parallel
110
0021-9991/90$3.00
Copyright 0 1990 by Academic Press, Inc.
All rights of reproduction in any form reserved.
 NEURAL COMPUTING OF DIFFERENTIAL EQUATIONS 111

algorithms for solving the finite difference equations. In Section II, the basic idea of
neural networks is reviewed, and general continuous or discrete neural minimiza-
tion algorithms are introduced. As an example, in Section III, the differential equa-
tion U’ =f(u) is considered to show how the neural minimization algorithms can
solve the equation, and the result of numerical simulation of the equation U’ = u is
described. In Section IV, general continuous and discrete neural algorithms for
solving a wide range of complex partial differential equations are derived. In
Section V, implementation schemes of neural algorithms utilizing high-capacity
optical interconnecting devices are described. Characteristic features of the neural
algorithms compared with conventional algorithms are discussed in Section VI.

II. NEURAL MINIMIZATION ALGORITHMS

A. Collective Computation of Neural Networks

Neural networks consist of individual processors and interconnections between
the processors. As an example, the schematic representation of a simple neural
network is drawn in Fig. 1. Each processor is called as a neuron which has an
analogy in biological neural systems.Each neuron can have two different states, i.e.,
on and off which are represented by binary numbers 1 and 0 (or 1 and - 1). The
operation of all the neurons is the same.Typically, each neuron sums all the signals
coming from all the other neurons through the weighted interconnections,
thresholds the summed signal to 0 or 1 (or - 1 or l), and changes its state
according to the thresholded output. This operation occurs at every neuron in the
network simultaneously or randomly. Therefore, if the total number of neurons in
the network is large, the state of the whole network consisting of the individual
neuronal state changes dynamically, and the network exhibits cooperative effects.
This kind of cooperative effect is very common in statistical physics.
The neural network described above can be utilized for computation. The binary
states of the neurons can be identified as a binary representation of some variables.
The interconnection strenghts between the neurons may represent the information

FIG. 1. Schematic diagram of a simple neura! network. T,,, Tik, and Tjk are connection strengths
between the pairs of neurons (i, j), (i, k), and (j, k).

581/91/l-S
112 LEE AND KANG

of the specific problem. Then, an initial trial state for the network representing the
initial trial solution may converge to the final state of the network according to the
neural dynamics, and it may give the solution of the problem. This method of com-
putation is based on the collective interaction between the neurons, and it exhibits
a high degree of parallelism. Another characteristic of the neural network is that the
processing of each neuron is extremely simple, however, the large number of
neurons and interconnections yields enormous computational power.

B. Continuous Neural Minimization Algorithm

The Hoplield model has been successfully applied for solving combinatorial
optimization problems such as the Hitchcock problem and the travelling salesman
problem. The Hopfield model [6] is based on the nonlinear dynamic interaction
between globally interconnected neurons. The states of individual neurons are
specified by their outputs Vi which range between 0 and 1. The dynamics of
neurons, i.e., updating rules of the neuronal states, in the Hopfield model can be
summarized as follows. In the continuous model, neurons change their states
according to the following equations of dynamics

dU,/dt = 1 T, V,, (2.1)

vi = g( uih (2.2)
where t is the continuous time which corresponds to the updating parameter, Tti is
the interconnection strength, and g(x) is a nonlinear function whose form can be
taken to be

g(x) = (lP)Cl + tanh(x/xo)l, (2.3)

which is a monotonically increasing function bounded between 0 and 1. x0 is a
constant. Hoplield has shown that if T, = Tji, neurons in the continuous model
always change their states in such a way that they minimize an energy function
defined by

E= -(1/2)~~ T,ViV,, 12.4)

i j

and stop at minima of this function. The updating rule represented by Eqs. (2.1),
(2.2), and (2.3) for minimizing the energy function (2.4) is highly parallel, and it has
been implemented by utilizing optics [7).
The Hoplield model described above can be applied only to minimization of
quadratic energy functions which have symmetric T matrices. However, the
Hopfield model for arbitrary energy functions has been investigated recently [8].
Consider an energy function for a minimization problem that can be written by
E = F( Vi), (2.5)
 NEURAL COMPUTING OF DIFFERENTIAL EQUATIONS 113

where F is a non-singular and bounded function of variables Vi, and the partial
derivatives with respect to Vi are assumed to be well defined. E is assumed to be
always positive or zero, which is valid because E is bounded. The time evolution
of the energy function defined by Eq. (2.5) is given by

dE/dt = 1 (aF/a Vi)(dVi/dt). (2.6)

If the updating rule is defined by

dUifdt = -aF/aVi, (2.7)
vi = h( Vi), (2.8)
where h(x) is a monotonically increasing function bounded between 0 and 1, and
Ui is the intermediate variable, it can be proved that Eqs. (2.7) and (2.8) minimize
the energy function as follows. From Eq. (2.8), we have
dVJdt = h’( Ui)(dUi/dt), (2.9)
where h’ is the derivative with respect to x. h’(x) is always positive or zero because
h(x) is a monotonically increasing function. If the updating rule represented by
Eqs. (2.7) and (2.8), and Eq. (2.9) is used in Eq. (2.6), we have

dE/dt = -c h’( U,)(dU,/dt)*. (2.10)

Equation (2.10) is always negative or zero. Therefore, the change of the energy
function in time according to the updating rule Eq. (2.7) and (2.8) guarantees mini-
mization of the energy function.

C. Discrete Neural Minimization Algorithm

The synchronous discrete neural algorithm for minimizing a wide range of energy
functions has been developed recently [9]. The energy function is assumed to be an
arbitrary type of polynomial function of the state variables. Real binary variables
having values 1 and - 1 are considered as state variables. The energy function can
be described as

E=f’({B,, B,, .... B,}), (2.11)

where B’s are the state variables and the total number of state variables is N.
Partially synchronous minimization is considered for the most general case.
Totally synchronous or totally asynchronous minimizations are specific examples of
the general case.Assume that, at each step, M state variables are selectedrandomly,
and minimization is carried out by updating the M state variables simultaneously
and leaving all the other state variables unchanged. M can be any integer from 1
to N, and the minimization algorithm becomes totally asynchronous or totally
114 LEE AND KANG

synchronous if M is equal to 1 or N. A set P is defined to consist of the indices for

the selected state variables. Another set P’ = { 1, 2, .... N} - P is defined to represent
the indices of the state variables which are not changed. The updated state variables
BI and the change of the state variables AB, satisfy the relation

B; = Bi + ABi, (2.12)
where ie P. The updated state variables are also binary variables having values 1
and - 1. Therefore, the possible values of AB, are
ABi= -2, 0, 2. (2.13)
The incremental change in energy AE due to the updated state variables given by
Eq. (2.12) is considered to develop an algorithm which minimizes the energy
function described by Eq. (2.11). AE is defined as

AE=EC(Bj, Bi}l -EC(Bi, Bi}], (2.14)

where ie P and je P’, and utilizing Eq. (2.12), it becomes
AE=E[{Bi+ABi,Bj}]-E[(Bi,Bj}]. (2.15)
The first term in the right-hand side of Eq. (2.15) can be expanded as a Taylor’s
series in several variables because E is a polynomial of the state variables. There-
fore, the incremental energy changes AE can be written as

AE= c 1 ... 1 (l/m!)[ABn . ..AB.] D[B,, . ..B.] E, (2.16)

m=l ilEP imeP

where D [ Bi, ... B,] E is a partial derivative with respective to the state variables
Bi, . . . B, at ABi= 0 for all iE P. The total number of terms in the summation of
Eq. (2.16) is finite because E is a polynomial.
To reduce the products of changes in Eq. (2.16) to linear forms, the following
relations are derived. For an arbitrary state variable B and a positive integer n,
(AB)n satisfies
(AB)“= (-2B)“-’ AB, (2.17)

which can be proved as follows. If AB is zero, Eq. (2.17) is satisfied automatically.

If AB is 2, B and B’ should be - 1 and 1 according to Eq. (2.12). In this case, the
right-hand side of Eq. (2.17) becomes the same as the left-hand side; i.e.,
[(-2)(-l)]“-’ [2] = 2”= (AB)“. (2.18)

On the other hand, if AB is -2, B and B’ should be 1 and - 1 according to

Eq. (2.12). Therefore, the right-hand side of Eq. (2.17) becomes
[(-2)(l)]“-’ [ -2]= [ -2]“, (2.19)
 NEURAL COMPUTING OF DIFFERENTIAL EQUATIONS 115

which is the same as the left-hand side of Eq. (2.17) in the case of AB= -2.
Consider next a product of changes given by AAB,, ... AB, with an arbitrary
coefficient A. Assume that m > 1. The product can be written as
AABi, ~~~AB,=~A)(1/2)[-{S(A)ABil-AB,~~~~AB,)*
+ (ABil)* + (ABiT*..AB,)*]p (2.20)
G IAI(lP)C(ABil)* + (AB,..~ABim)21, (2.21)
where S(A) is the sign of A. The first term in Eq. (2.20) is always negative or zero,
and Eq. (2.21) follows. The second term in Eq. (2.21) is a product of smaller
number of changes than the left term in Eq. (2.20). Therefore, this technique can be
applied iteratively to reduce the product of changes in Eq. (2.20) to a sum of
individual changes, and it is given by
m-1
AAB,,
j=l
c 2-~(ABii)2i+2-(m-1)(ABi,)2(m-‘)
1
. (2.22)

Equation (2.22) can be symmetrized by considering m cyclic orderings of indices

{ il, .... im}. Equation (2.22) now is given by

+2-b- l)(ABik)*+‘)
1. (2.23)

If Eq. (2.17) is utilized, Eq. (2.23) can be written as

AABil .**ABi,,,< IAl Z(m) ‘f Bik ABik, (2.24)

k=l

where

Z(m)= _ 4
( >[
m~12(2j-j-1)+2(*(m-~L-m)
j=l 1.
(2.25)

If Eqs. (2.24) and (2.25) are used in Eq. (2.16), the incremental energy change is
given by
dzx -C Gi ABE, (2.26)

m=l ilEP

x PCW4, . ..Bim]EI)+D[Bi]E]* (2.27)

116 LEE AND KANG

From Eq. (2.26) it is clear that if

Gi AB,aO for all i E P, (2.28)
the incremental energy change is always negative or zero. Therefore, Eq. (2.28)
updates the state variables in such a way that it minimizes the energy function in
the limit of many iterations. For positive G,, Eq. (2.28) is satisfied if ABi is positive
or zero. If Bi is equal to 1, Bj should be equal to 1 becauseABi = 0 is the only solu-
tion. However, if Bi is equal to -1, BI should be 1 because this makes the
incremental energy change more negative than using the condition AB,=O. There-
fore, if Gi is positive, the updated value becomes 1 which is the same as the sign
of the value Gi. If Gi is negative, the above argument can be applied to show that
the updated value for B,! becomes - 1. This is the same as the sign of Gi. If Gi is
zero, B; can have any values. Summarizing the above result, the updating rule can
be written as
B; = T(G,) for all i E P, (2.29)
where T is a unit step function defined by T(x) = 1 if x > 0 and T(x) = - 1 if x < 0.
Equation (2.29) is the general discrete neural algorithm which minimizes energy
functions consisting of arbitrary types of polynomials of the state variables in a
partially synchronous way.

III. CASE STUDY FOR SOLVING DIFFERENTIAL EQUATIONS

A. Continuous Algorithm for u’ = f (u)

A simple example is considered to explain how neural minimization algorithms
described in Section II can be utilized to solve differential equations numerically.
The differential equation is chosen as
du/dx=f(u), (3.1)
where f is a polynomial function of u. The finite difference equation for Eq. (3.1) is
given by

(U,+1- U,-,)/2h=f(U,), (3.2)

where h is the mesh size for discretization, and U, is the discrete variable corre-
sponding to the mesh index a. For the continuous neural algorithm, a binary
representation for the variable U, may be given by

u,= 5 2s-Iva/as- {(1/2)[2K- l]}, (3.3)

s= -K

where K and K’ are positive integers, and V,, are binary variables.
 NEURAL COMPUTING OF DIFFERENTIAL EQUATIONS 117

The energy function for the continuous model is defined by

where A is a positive constant and the updating rule is derived from the time
derivative
dE/dt=x 2A[(U,+,- ua- IF9 -f(UJl wJ,+ I/aVbJPh
abs

- (8 u, - ,/a Vbs)/2h - af( ua)/a vbs > (dVbs/dt). (3.5)

The expression of the intermediate variables for the updating rule given by Eq. (2.7)
becomes

dW,,/dt=-x2A[(U,+,- ua-d/2h-f(Uo)i wa+l/avbs)/2h

-~au,-,/avb~)/2h-af(u,)/avb~} (3.6)
= -2A([(U,- u,-,)/2h-+f-(Ub-,)l/2h
- [tub+,- Ub)/2h - f ( Ub + 1) IPh
- [tub+, - ub-mh-wb)] (df(ub)/dub)) taubiaVbs). (3.7)

Therefore, the continuous neural agorithm for solving Eq. (3.1) consists of
dW,,/dt= -2A{(1/2h)‘(-U,,, +2ub- ub-2)+(1/2h)[f(Ub+l)

-f(“b-l)-(ub+,- ub-,)(dfub)/dUb)l

+f(ub)(df(Ub)/dUb)1) taubiaVbs). (3.8)

vbs = g( wbs), (3.9)
where g is the threshold function given by Eq. (2.3), and the solution is given by the
v,, (t = cc ).

B. Numerical Simulation
Numerical simulation of the continuous neural algorithm described in Section III.
A was carried out to explain in detail how it works. The differential equation is
chosen as
du/dx = u. (3.10)
The finite difference equation for Eq. (3.10) is given by

(Ua+1- U,- ,)/2h = U,. (3.11)

The binary representation for the variables U, described in Eq. (3.3) was utilized.
118 LEE AND KANG

The energy function is given by

E=CAC(U,+,-U,~,)/2h-U,12, (3.12)
0
and the updating rule can be read from Eqs. (3.8) and (3.9) as follows:
dW,,/dt = -2A((1/2h)’ (- Ub+2 + 2Ub - Ub&2) + U&J} (au,/av,,). (3.13)
Vbs= g( Wbs)- (3.14)
Consider the differential equation (3.10) in the interval 0 < x < 1. The initial value
is given by u(0) = U,. The unit interval is discretized with a step h. The discretized
variables are U, , U2, .... U,, where hiV = 1. Then, the discretized form of Eq. (3.13)
in time t without thresholding is given by
U,(k+l)=U,(k)+24tA{[(1/2h)*(U,+,(k)-2&,(k)

+ U,-,(k)) - U,(k)1 1, (3.15)

where b = 1, .... N, k denotes the number of iterations in time, and At is the
incremental time step for the iteration. However, Eq. (3.15) contains undefined
quantities U- r, U, + r, and U, + 2. The values for these variables at each iteration
are obtained by considering Eq. (3.11)
U-,(k) = U,(k) - 2hU,,, (3.16)
u,, I(k) = u,- I(k) + 2hU,W), (3.17)
U,+Ak) = U,(k) + 2hCU,- I(k) + 2hU,(k)l. (3.18)

3.
2.7182 k=lSO
k=lOO

k=75
2.

k=50

k=25
1. k=l

FIG. 2. Convergence of the initial trial solution U, = 1 for i = 1 to 100. The solution was stabilized
after 170 iterations.
 NEURAL COMPUTING OF DIFFERENTIAL EQUATIONS 119

Equations (3.15)-( 3.18) are utilized to obtain the solution for the differential
equation.
Computer simulation based on Eqs. (3.15~(3.18) has been performed. The
parameters were chosen as h = 0.01, N = 100, and AtA = 0.0001. Here, AtA was
chosen as a maximum value satisfying the condition that the final converged solu-
tion does not depend on AtA characterizing discretization of the differential equa-
tion (3.13) with respect to the parameter t. For simplicity, the algorithm without
thresholding was considered, and AT&T PC6300 was used. The initial value was
U, = 1. Two sets of values for Ui, i = 1 to N, at the k = 0 iteration step, i.e., initial
trial solutions, were selected to show that both trial solutions converge to the
correct solution. Figure 2 shows convergence of the trial solution given by Ui = 1
for i= 1 to N. Another trial solution defined by Ui= 1. 5hi+ 1 for i= 1 to N is
shown to converge to the correct solution in Fig. 3.
C. Discrete Neural Algorithm
Derivation of the discrete neural algorithm for solving the differential equation
U’ = u is presented to illustrate the algorithm introduced in Section 1I.C. Totally
asynchronous algorithm, i.e., updating one state variable at a time but randomly,
is considered for simplicity. The energy function for the discrete model is given by
E=C C(Ua+,- ua-,)P- Kzl*, (3.19)

where we set A = 1. The binary representation of variables U, is chosen as

u,= i;. 2”-I@,+ 1)/2- {(1/2)[2K- 111, (3.20)
s= --K

FIG. 3. Convergence of the initial trial solution CT,=l.Sih+ 1 for i= 1 to 100 and h=O.Ol. In this
case, the initial trial solution is closer to the correct solution than the initial trial solution used in Fig. 2.
However, the solution was stabilized after 160 iterations.
120 LEEANDKANG

where K and K’ are positive integers, and B,, are binary variables having values 1
and - 1. The incremental change in energy due to the variable B,,y is

AE= [(t-J,+, - U,+,)/2h- UC-2”-’ ABC,]’

+ [(Uc+2”-’ ABC,- UC-,)/2h- U,p,]2
+ C(Uc+,- U,-2”-2 AB,,)/U- UC+,]’

- C(U,+, - UC+ ,)/2h - U,]’

- C(u,- Uc-JPh- U,-,12
- C(Uc+2 - Uc)Ph- Uc+J2
=2{ -(Ur+2- 2U,+ UC-,)(1/2h)‘+ UC} 2”-2 AB,
-2(1 +2(1/2h)2} (2”-2)2 B,, AB,,, (3.2:1)

which gives the intermediate variable for the discrete model as

%=wL+2- 2U,+ U,-2)(1/2h)2- UC} 2”p2

+ 2{1 +2(1/2h)2}(2”P2)2 B,,. (3.22)

Equation (3.22) together with the step thresholding function gives the discrete
neural algorithm.

IV. GENERAL FORMALISM FOR SOLVING DIFFERENTIAL EQUATIONS

A. General Algorithms
The general form of differential equations to be considered in this section is
represented by

Fi (x,, uj, d,,,uk, higher-order derivatives) = 0, (4.1)

where I, m = 1 to M, x1 are independent real variables, and i, j, k = 1 to N, uj are
dependent real variables, and d,u, are first-order partial derivatives, and there are
N functions indexed by i. The functions Fi are assumed to be non-singular, and
partial derivatives with respect to x,, uj, d,,,uk, and higher-order derivatives are
assumed to be well defined. The finite difference method is introduced to solve
the differential equation (4.1). The finite domain in M dimensional space for
independent variables is discretized with a unit volume h”. A discrete index vector a
with components a(l), I= 1 to M, is introduced to describe such a space. Each
component a(1) represents the discrete coordinate for the individual variable xl.
 NEURAL COMPUTING OF DIFFERENTIAL EQUATIONS 121

Then, the discretized quantities corresponding to the continuous quantities can be

written as
xla = xl0 + a(W4 (4.2)
U,a = +G(ah (4.3)
DmUka= WWUJ~,,+,~- U,c,a-,,,,I
=(l/W~ kL+m~,d,-m,,J U,,, (4.4)
b

Dim u,, = (D,(Dm ud),

=D,W~,a+,,,- &,a-,W)

=(1/2h)*x {6.+,‘+,‘,b-6.+,‘-,‘,b
b

-6 a-d+I’,b+ 6a-m’-,‘,b) ‘kb, (4.5)

where 6 is a unit matrix, xIo is the initial value for th coordinate xl, m’, and 1’ are
M-dimensional vectors satisfying the conditions m’(m) = 1 and m’ = 0 otherwise
and l’(1) = 1 and I’ = 0 otherwise. The finite difference equation for the differential
equation (4.1) is then given by

Fia(X/a9 uja 3Dm u,ca2 higher-order discrete derivatives)

= FB( Vi,) = 0. (4.6)
Continuous algorithm is considered first. The variables Ujb are continuous real
variables. However, we may use different types of number representation of the real
variable. Let one of the possible representations of the real number be given by

ujb = R(Bjbr)> (4.7)

where R is the representation function, r denotes the bits, and Bjbr are bit variables
having two binary values 0 and 1. If Eq. (4.7) is used, the finite difference equation
(4.6) becomes
F,( X,, , Up, D, U,, , higher-order discrete derivatives)
=F;a(Ujb)=F;(B,b,)=O. (4.8)
To exploit a continuous algorithm, continuous variables Vjbr which are bounded
between 0 and 1 are substituted with Bib, in Eq. (4.8). Then, Eqs. (4.7) and (4.8)
become
‘jb = R( ‘jbrh (4.9)
Fia (Xla 7 uja 3Dm u,a 3 higher-order discrete derivatives)
=F:,(Ujb)=F~(Vjb,)=O. (4.10)
122 LEE AND KANG

Equation (4.10) can be solved by utilizing a minimization algorithm described in

1I.A. An energy function for Eq. (4.10) is introduced as

E( V,,,) = A c FL*, (4.11)

ia

where A is a positive constant. If the energy function given by Eq. (4.11) is mini-
mized with respect to Vibr, the solution of the differential equation is given by the
values Vi& co) which minimize Eq. (4.11). The time derivative of Eq. (4.11) is given
by

dE( Vjbr(t))ldt = 1 (aE/aVi~,)(dVi~,ldt)’ (4.12)

ias
From Eqs. (2.7) and (2.8), the minimization algorithm is given by
d W,Jdt = - aEli V,, , (4.13)
Via, = G( WiasL (4.14)

where W,, are intermediate variables, and G is a monotonically increasing

thresholding function bounded between 0 and 1.
Consider the right-hand side of Eq. (4.13). It can be given by

aqav,, = A c (aF;,'/av,,), (4.15)

kc

and, from Eq. (4.10),

aFi?av,, = 1 (aFl,‘,/aUjb)(aUjb/a Vias), (4.16)

ib

and
(aF,,2,1aujb)=aF:,faujb
+ c [aF:cia(DmuF)i(a(Dm up)/(aujbl
mP

+ C caF:,/a(~,,~,,)i~a(~,,u,)iau,~i
m[P

+ higher-order terms. (4.17)

If Eqs. (4.4) and (4.5) are used, we have

(4.18)
 NEURAL COMPUTING OF DIFFERENTIAL EQUATIONS 123

and

am,u,,w,,
=ww* c {Se+m’+,‘,e-6e+m,-r,,e
-6cd,,.,.+6,-*,-,,,,>(aup,lau,b)
=(1/2h)2 ~jp{6c+m'+l',b-bc+m'--l',b

-6 e--m'+,'.b+~c--m'-,I',b).
(4.19)

Substituting Eqs. (4.18) and (4.19) into Eq. (4.17), it becomes

+I CaF-:,la(D,u,)1(1/2h)6jp(Bc+,‘,b-6,-,’,b)
mP

+ C CaF~,/a(D,,“,)1(1/2h)2 djp{bc+mr+l’,b
mlP

-6 -6 e--m’+l’,b+~c-m’-,I’,b}
e+m’-l’,b

+ higher-order terms (4.20)

= aF:,laujb

+I [aF:,/a(D,Uj~)l(1/2h>(S,+,:b-6,~,:b)
m

+I [aF:,/a(D,,U,)1(1/2h)2 &+d+l’,b
ml

-6 c+m’-f’,b -6 e-m’+I’,b+6e-m’-,‘,b}

+ higher-order terms. (4.21)

From Eqs. (4.16) and (4.21), Eq. (4.15) becomes

aE/av,, = A 1 aF:,/auib
jbke

+c [aF~,/a(D,uj~)l(1/2h)(6,+,:b-6,-,:b)
m

+I CaF:,/a(D,,Uj=)1(1/2h)’ (&+m’+,‘,b
ml

-6 c+m’-t’,b -6 c~m’+I’,b+Be--m’--[‘,b)

+ higher-order terms (aUjb/aVi,,,) (4.22)

1
124 LEE AND KANG

= Ac 8F:,laU,
kc

+I (‘/“)(‘,~,,,,-‘,+,,,,)C’F~,I’(‘,’i~)l
m
+c w2~)* (Sa--m’~I’,c-6a~m,+r,,c
ml
-6 P+m,-I.,r+~n+m,+,,,c)CaF:,la(o,,u,)l

+ higher-order terms (au,/8 Vi,,), (4.23)

where Eq. (4.9) was used to sum over j and b, and a and c in the unit matrices 6
were rearranged. Utilizing the relations (4.4) and (4.5) in Eq. (4.23), the algorithm
for solving the differential equation is finally given by

dW,Jdt = -A 1 a(F;:,)/Z’,
k

+ 1 o,,Ca(F:,)/a(o,,u,)l
ml
+ higher-order terms (8 U,/a Vi,,s), (4.24)

and

J’,, = ‘3 W,, ). (4.25)

Discrete formalism for solving differential equations is based on the discrete

neural minimization algorithm developed in Section 1I.C. The finite difference equa-
tion for the differential equation (4.1) is given by Eq. (4.6). However, the binary
representation of the variables U, is achieved by using binary variables having
values 1 and - 1. For example, Eq. (3.20) can be used. Therefore, the energy
function for the discrete algorithm is given by

Wja,) = 1 U&J*, (4.26)

ia

where Bjbn are binary variables having values 1 and - 1. The discrete algorithm can
be obtained by using Eqs. (2.27) and (2.29).

B. Boundary Conditions
Dynamical equations, i.e., updating rule, which yield solutions for differential
equations have been developed. However, the initial or boundary conditions of the
 NEURAL COMPUTINGOFDIFFERENTIALEQUATIONS 125

differential equations have not been considered. In this section, the method of
incorporating the initial or boundary conditions in the updating rule is discussed.
First, it is shown that initial conditions can be considered as boundary condi-
tions. As described in Section III.B, consider a first-order differential equation of
one independent variable x. The initial condition may be written as u(xO) = ZQ,,
where u is the dependent variable and x0 is the initial point. In this case, the final
point x1 should be specified to solve the differential equation numerically. There-
fore, if 0 and N+ 1, which correspond to discretized coordinates of x0 and x1 + h,
are selected as the boundary points, the variables of the problem are U(l),
WI, .**,U(N) and the boundary conditions are given by U(0) and U(N+ 1). In this
case, U(N+ 1) is not a fixed value but varies subject to the original finite difference
equation. Therefore, U(N+ 1) can be written as a function of U(O), U(l), .... U(N).
Boundary conditions including initial conditions can be incorporated as follows.
First, select the boundary for a problem. Then, the domain surrounded by the
boundary is discretized. The boundary values are assigned to the discrete variables
for the boundary points. However, the dynamical equations presented in
Section 1V.A are chosen only for the variables inside the boundary. The values for
the boundary variables are supplied by utilizing the finite difference equations at the
boundary at each iteration.

V. IMPLEMENTATION OF NEURAL ALGORITHMS

In this section, possible implementation schemes of the neural algorithms are

presented. As discussed in Section II.A, one of the most important characteristics in
the neural network is the interconnection between neurons, because the individual
neuron performs a simple operation. Therefore, the collective computational
properties emerge as a result of large number of neurons and interconnections
between the neurons. Electrical implementation of a large number of interconnec-
tions, for example, lOI* interconnections,, is very difficult. However, optical
implementation is very attractive to overcome the difficulties encountered in electri-
cal implementation.
Optical implementation of the neural algorithms for solving differential equations
is based on interpreting the algorithms equations (3.8) and (3.22) as follows. Con-
sider a two-dimensional optical wave propagating in free space. At each iteration,
the discretized independent variable V,, (continuous model) or B,, (discrete model)
is represented by the light amplitude in a two-dimensional optical wave at each
resolution point which is identified as a neuron. Then, at each neuron, the inter-
mediate variable W,,, is obtained by simply multiplying and summing I’,, (or B,,)
from other neurons, i.e., neurons interconnected to the neuron represented by bs’,
according to Eqs. (3.8) or (3.22). Here, interconnections are achieved by optical
volume holograms. Multiplication and summation can be implemented optically or
electronically. The updated values for the neurons are obtained by thresholding the
intermediate variables. Thresholding can be performed by using nonlinear optical
126 LEE AND KANG

phenomena [ 101 or an electronic method. Finally, the updated values for the
neurons represented by a two-dimensional optical wave are fed back to the incident
port of the optical system. Gain is included in the feedback loop to compensate
the loss. The solution of the differential equation is obtained when the iteration
converges. The above scheme of optical implementation is illustrated in Fig. 4.
Optical implementation of the interconnection based on volume holography
utilizing photorefractive crystals has been extensively investigated [4]. The motiva-
tion for using volume holograms comes from their ability to store information in
three dimensions. Using volume holographic technique, it is possible to impress a
grating pattern into such a crystal so as to transfer a light signal from an input
point to another point in an output plane. Another input-output connection can
similarly be implemented by a second grating oriented inside the crystal at a dif-
ferent angle from the first one. The maximum number of connections that can be
specified if a volume hologram is used is upperbounded by the degrees of freedom
available in the volume of the crystal which is equal to V,/A3, where VH is the
volume of the hologram and 2 is the optical wave length. As a numerical example,
consider an optical wave with wavelength lpm, a 1 cm square input plane. and 1 cm
cube volume holograms. Then, the total number of neurons which can be processed
in the system is lo’, and the total number of interconnections becomes 10”.
The proposed optical system fully exploits the advantages of optical parallel com-
puting. It makes use of a two-dimensional optical wave to represent discretized
independent variables. High-capacity volume holograms are used to interconnect a
huge number of variables. Furthermore, the system is one of the most parallel
systems that can be implemented to solve the finite difference equations. The total

MULTIPLICATION
AND SUM
INPUT
. . . OUTPUT
. v . “‘, ” . . . .. . . . . . .
@&k) \\lr6kk’ . ydk+” .
. r

INTERCONNECTlON THRESHOL3
(VOLUME HOLOGRAM)
1.

FEEDBACK

FIG. 4. Optical system for solving differential equations. Mirrors and beam splitters for the optical
wave arc not drawn.
 NEURAL COMPUTING OF DIFFERENTIAL EQUATIONS 127

number of interconnections may be reduced in actual implementation due to the

Bragg cone effect, higher-order cross-talk effects, and material quality. However,
10” interconnections seem to be feasible.

VI. DISCUSSIONS

The main purpose of this paper is to introduce the concept of neural networks
and to develop neural algorithms for solving differential equations. The detailed
analysis on the algorithms will be published separately. However, general features
of the neural algorithms will be discussed in this section. The remarkable collective
computational properties such as recognition from partial input, robustness, and
error-correction capability have been demonstrated recently. As an example,
Hoplield introduced associative memory in the neural network described in
Section 1I.A. The memory vectors @“I, where i represents the neurons and m
describes the number of memory vectors, are stored in the weight of the inter-
connections as T.. = C BP’B!“’ summed over the memory vectors. The computa-
tional function o/the Hdplieid neural network is to reconstruct the whole original
memory given an initial partial memory. In simulations, correct convergence was
obtained for the total number of four memory vectors in the case of 30 neurons
when the initial trial memory vector differed from the original memory vector by
seven bits [6]. Therefore, the radius of convergence of the neural network is in
general very large. This property is very important in solving differential equations.
In conventional algorithms, a good initial guess of the solution is essential in
solving differential equations effectively. However, the radius of convergence in
neural algorithms is expected to be very large, and thus, selection of an initial trial
solution is not so sensitive. This has been demonstrated in the results of numerical
simulation shown in Figs. 2 and 3.
Another important characteristic of the neural network is nonlinear dynamics
introduced by nonlinear thresholding. This property is expected to yield a large
processing gain in neural computation. As an example, for the neural associative
memory described above in this section, the number of iterations which yielded the
correct original memory was approximately 5. Therefore, it is expected that the rate
of convergence in neural algorithms is much faster than the conventional algo-
rithms. The second property of neural algorithms which affects the rate of con-
vergence is the collective processing inherent in the neural networks. This is because
the greater the problem size, the more neurons participate collectively in solving the
differential equations. To illustrate this, consider a stiff differential equation. To
solve this equation, the number of mesh points should be very large so that dis-
cretization of the equation is accurate and smooth. This means the total time, i.e.,
the rate of convergence, required for solving the equation increases rapidly if the
conventional serial algorithms are used. In the neural algorithms proposed in this
paper, the equation can be solved by increasing the total number of neurons.
However, the time for solving the equation actually decreases even if the total
128 LEE AND KANG

number of neurons increases, becauseat each iteration all the neurons interact with
each other in parallel, i.e., collectively. This point has been demonstrated by con-
sidering the specific example described in Fig. 2. We solved the same problem
assuming all the same conditions except decreasing the total number of mesh points
between 0 and 1. Fifty mesh points instead of 100 mesh points were chosen, and h
was 0.02 in this case. The constant AtA was 0.0001. The result of the numerical
simulation is shown in Fig. 5. Comparing Figs. 2 and 5, it is clear that as the
number of neurons increase, the rate of convergence decreases.Specifically, the rate
of convergence for this example decreased by half when the number of neurons
increased twice. The third property which decreasesthe time for solving differential
equations is the optical feedback used in implementing the neural algorithms. This
means communication between processors, i.e., neurons, is performed by the fastest
physical method. Therefore, optical implementation reduces the computation time
even if the total number of iterations becomes very large.
In the above paragraph, the rate of convergence for the stiff differential equations
has been investigated by changing the total number of mesh points, i.e., neurons.
However, it is also important to study the rate of convergence for the stiff differen-
tial equations maintaining the same total number of mesh points and changing the
degree of stiffness. As an example, a differential equation du/dx = --Au was
investigated in the interval 0 <x < 1 with initial condition u(0) = 1, where ,? is a
positive constant and serves as a stiffness parameter. The updating rule for this case
can be obtained from Eq. (3.8)
U,(k+ l)= Ub(k)+2AtA{[(1/2h)* (U,+,(k)-2U,(k)
(6.1)

2.7182

2.

k=SO
k=l

I I >
0. 0.5 1. x
FIG. 5. Convergence of the same problem as described in Fig. 2 when the number of mesh points
was decreased to 50.
 NEURAL COMPUTING OF DIFFERENTIAL EQUATIONS 129

0. ~, I)
0.5 1. x

-1. -

-2.

FIG. 6. Convergence of the differential equation u’ = -0.5~. The converged solution was obtained at
k= 138.

which is the same as Eq. (3.15) except for the A* factor. The boundary conditions
are given by
U-,(k) = U,(k) + 2hMJ,,, (6.2)
U,, I(k) = U,- 1%) - 2hJU&), (6.3)
U,+,(k) = U,(k) - 2hA[U,- ,(k) - ZhLU,(k)]. (6.4)

FIG. 7. Convergence of the differential equation U’ = --u. The converged solution is not good
compared with Fig. 6.
130 LEEANDKANG

-2.J
FIG. 8. Convergence of the differential equation u’ = - 321.The converged solution was obtained at
k = 156. However, the solution looks very bad compared with Figs. 6 and 7.

Computer simulations based on Eqs. (6.1~(6.4) have been carried out for three dif-
ferent values of A =OS, 1, and 3. The other parameters were chosen as h =O.Ol,
N= 100, and AtA = 0.0001 for the three different values of A, which are the same
parameters as those used in Fig. 2. The same initial trial solution for three different
values of A were chosen as Vi(O) = 1, i= 1, .... 100. The numerical results are shown
in Figs. 68. Figure 6 represents the case of A= 0.5, and the algorithm converged at
k = 138. Figure 7 shows the result of A.= 1. The solution in this case converged at
k = 140. However, as shown in the figure the converged solution is not good com-
pared with the case of il= 0.5. When the stiffness parameter 1 was increased to 3,
the algorithm converged at k = 156 as shown in Fig. 8. However, this solution is
much worse than the case of A= 1. From this simple example, we see that the rate
of convergence and effectiveness of the proposed neural algorithms deteriorates
when the differential equations become stiffer and the total number of mesh points,
i.e., neurons, are kept the same.
Finally, the neural algorithms proposed in this paper have several advantages
over the conventional parallel algorithms. First, the neural algorithms are much
simpler than others. The only algebraic operations required in the neural algo-
rithms are addition and multiplication. Inverse or other complicated logic opera-
tions are not needed. Second, the neural algorithms are the most parallel, compared
with conventional parallel algorithms. For example, the algorithms for a hypercube
have limited parallelism due to a small number of interconnections. Simplicity in
implementation of the neural algorithms is the third advantage.

VII. SUMMARY
Highly parallel neural algorithms for solving finite difference equations have been
developed. They can be applied to a wide range of differential equations. Nonlinear
 NEURAL COMPUTING OF DIFEERENTIAL EQUATIONS 131

thresholding which is a characteristic of neural networks has been introduced in the

algorithms. Implementation of the algorithms can be realized by using electronic
processors if the order of the differential equations is low, because in this case the
number of interconnections is very small. However, if higher-order differential equa-
tions are considered, optical implementation is better. Finally, general features of
the neural algorithms for solving differential equations have been discussed.

ACKNOWLEDGMENTS

The authors acknowledge valuable comments by the reviewers. H. Lee would like to acknowledge
support by the National Science Foundation Grant No. EET-8810288 and the Center for Advanced
Technology in Telecommunications. We thank W. S. Baek for computer simulation.

REFERENCES

1. Physics Today, Special Issue: Computational Physics, October (1987).

2. J. J. HOPEIELD AND D. W. TANK, Biol. Cybernet. 52, 141 (1985).
3. M. TAKEDA AND J. W. GOODMAN, Appl. Opt 25, 3033 (1986).
4. H. LEE, X. Gu, AND D. PSALMS, .I. Appl. Phys. 65, 2191 (1989).
5. G. F. GARRIERAND C. E. PEARSON, Parfial D#ereniial Equations: Theory and Technique (Academic
Press, New York, 1976).
6. J. J. HOPFIELD, Proc. Nat. Acad. Sci. 79, 2554 (1982).
7. D. PSALTLSSAND N. FARHAT, Opt. Let?. 10, 98 (1985).
8. C. KOCH, J. MARR~QUIN, AND A. YUILLE, Proc. Nat. Acad. Sci. 83, 4263 (1986).
9. H. LEE, Technical Digest of the Topical Meeting on Optical Computing, Salt lake city, Utah,
Feb. 27-Mar. I, 1989.
10. H. M. GIBBS, Optical B&ability: Controlling Light with Light (Academic Press, San Diego, 1985).