Quadratic-Type Lyapunov Functions for 
Competitive Neural Networks with 
Different Time-Scales 
Anke Meyer-Bise 
Institute of Technical Informatics 
Technical University of Darmstadt 
Darmstadt, Germany 64283 
Abstract 
The dynamics of complex neural networks modelling the self- 
organization process in cortical maps must include the aspects of 
long and short-term memory. The behaviour of the network is such 
characterized by an equation of neural activity as a fast phenom- 
enon and an equation of synaptic modification as a slow part of the 
neural system. We present a quadratic-type Lyapunov function for 
the flow of a competitive neural system with fast and slow dynamic 
variables. We also show the consequences of the stability analysis 
on the neural net parameters. 
I INTRODUCTION 
This paper investigates a special class of laterally inhibited neural networks. In 
particular, we have examined the dynamics of a restricted class of laterally inhibited 
neural networks from a rigorous analytic standpoint. 
The network models for retinotopic and somatotopic cortical maps are usually com- 
posed of several layers of neurons from sensory receptors to cortical units, with 
feedforward excitations between the layers and lateral (or recurrent) connection 
within the layer. Standard techniques include (1) Hebbian rule and its variations 
for modifying synaptic efficacies, (2) lateral inhibition for establishing topographical 
organization of the cortex, and (3) adiabatic approximation in decoupling the dy- 
namics of relaxation (which is on the fast time scale) and the dynamics of learning 
(which is on the slow time scale) of the network. However, in most cases, only com- 
puter simulation results were obtained and therefore provided limited mathematical 
understanding of the self-organizating neural response fields. 
The networks under study model the dynamics of both the neural activity levels, 
338 A. MEYER-BSE 
the short-term memory (STM), and the dynamics of synaptic modifications, the 
long-term memory (LTM). The actual network models under consideration may be 
considered extensions of Grossberg's shunting network [Gro76] or Amari's model 
for primitive neuronal competition lamaS2]. These earlier networks are considered 
pools of mutually inhibitory neurons with fixed synaptic connections. Our results 
extended these earlier studies to systems where the synapses can be modified by 
external stimuli. The dynamics of competitive systems may be extremely complex, 
exhibiting convergence to point attractors and periodic attractors. For networks 
which model only the dynamic of the neural activity levels Cohen and Grossberg 
[CG83] found a Lyapunov function as a necessary condition for the convergence 
behavior to point attractors. 
In this paper we apply the results of the theory of Lyapunov functions for singularly 
perturbed systems on large-scale neural networks, which have two types of state 
variables (LTM and STM) describing the slow and the fast dynamics of the system. 
So we can find a Lyapunov function for the neural system with different time-scales 
and give a design concept of storing desired pattern as stable equilibrium points. 
2 
THE CLASS OF NEURAL NETWORKS WITH 
DIFFERENT TIME-SCALES 
This section defines the network of differential equations characterizing laterally 
inhibited neural networks. We consider a laterally inhibited network with a deter- 
ministic signal Hebbian learning law [Heb49] and is similar to the spatiotemporal 
system of Amari [Ama83]. 
The general neural network equations describe the temporal evolution of the STM 
(activity modification) and LTM states (synaptic modification). For the jth neuron 
of a N-neuron network these equations are: 
N 
5:j -- -ajxj + E Dijf(xi) + B151 (1) 
i=1 
where x1 is the current activity level, a I is the time constant of the neuron, Bi is 
the contribution of the external stimulus term, f(xi) is the neuron's output, Dij is 
the.lateral inhibition term and Yi is the external stimulus. The dynamic variable 
Sj represents the synaptic modification state and ly121 is defined as lyl 2 = yTy. 
We will assume that the input stimuli are normalized vectors of unit magnitude 
lyl 2 = 1. These systems will be subject to our analysis considerations regarding the 
stability of their equilibrium points. 
3 
ASYMPTOTIC STABILITY OF NEURAL 
NETWORKS WITH DIFFERENT TIME-SCALES 
We show in this section that it is possible to determine the asymptotic stability of 
this class of neural networks interpreting them as nonlinear singularly perturbed 
systems. While singular perturbation theory, a traditional tool of fluid dynamics 
and nonlinear mechanics, embraces a wide variety of dynamic phenomena possesing 
slow and fast modes, we show that singular perturbations are present in many 
Quadratic-type Lyapunov Functions for Competitive Neural Networks 339 
neurodynamicM problems. In this sense we apply in this paper the results of this 
valuable analysis tool on the dynamics of laterally inhibited networks. 
In [SK84] is shown that a quadratic-type Lyapunov function for a singularly per- 
turbed system is obtained as a weighted sum of quadratic-type Lyapunov functions 
of two lower order systems: the so-called reduced and the boundary-layer systems. 
Assuming that each of the two systems is asymptotically stable and has a Lyapunov 
function, conditions are derived to guarantee that, for a sufficiently small pertur- 
bation parameter, asymptotic stability of the singularly perturbed system can be 
established by means of a Lyapunov function which is composed as a weighted sum 
of the Lyapunov functions of the reduced and boundary-layer systems. 
Adopting the notations from [SK84] we will consider the singularly perturbed system 2 
c=f(x,y) xBxCR n (3) 
e=g(x,y,e) yBy cR m,>0 
(4) 
We assume that, in Bx and By, the origin (x - y -- 0) is the unique equilibrium point 
and (3) and (4) has a unique solution. A reduced system is defined by setting e = 0 in (3) 
and (4) to obtain 
ft=f(x,y) (5) 
0 = g(x,y, 0) (6) 
Assuming that in Bx and By, (6) has a unique root y = h(x), the reduced system is 
rewritten as 
* = f(x, h(x)) = fr(x) 
A boundary-layer system is defined as 
(7) 
Oy 
Or = g(x, y(v),0) (8) 
where r = t/e is a stretching time scale. In (8) the vector x 6 R n is treated as a fixed 
unknown parameter that takes values in Bx. The aim is to establish the stability properties 
of the singularly perturbed system (3) and (4), for small , from those of the reduced system 
(7) and the boundary-layer system (8). The Lyapunov functions for system 7 and 8 are of 
quadratic-type. In [SK84] it is shown that under mild assumptions, for sufficiently small 
, any weighted sum of the Lyapunov functions of the reduced and boundary-layer system 
is a quadratic-type Lyapunov function for the singularly perturbed system (3) and (4). 
The necessary assumptions are stated now [SK84]: 
1. The reduced system (7) has a Lyapunov function V: R n - R+ such that for all 
xBx 
(vxv(x)) T fr(x) < > 0 (9) 
where b(x) is a scalar-valued function of x that vanishes at x = 0 and is different 
from zero for all other x  Bx. This condition guarantees that x = 0 is an 
asymptotically stable equilibrium point of the reduced system (7). 
2The symbol Bx indicates a closed sphere centered at x = 0; By is defined in the same 
Way. 
340 A. MEYER-BSE 
2. The boundary-layer system (8) has a Lyapunov function W(x, y): 1 n x 1 TM - 
R+ such that for all x  Bx and y  By 
C 2 
(VyW(x,y))Tg(x,y,0)_<-- aq5 (y-h(x)) (a >0 
(10) 
where b(y- h(x))is a scalar-valued function (y- h(x)) e 1 TM that vanishes 
at y = h(x) and is different from zero for all other x  Bx and y  By. This 
condition guarantees that y -- h(x) is an asymptotically stable equilibrium point 
of the boundary-layer system (8). 
3. The following three inequalities hold Vx 6 Bx and Vy 6 By: 
b.) 
c.) 
2 
(VxW(x, y))Zf(x, y) _< cb (y - h(x)) + ca;b(x)b(y - h(x)) (11) 
(V:V(x))T[f(x, y) - f(x, h(x))] </;/,(x)qS(y - h(x)) (12) 
(VyW(x, y))T [g(x, y, e) -- g(x, y, 0)] 
(y- h(x)) 
+ eKa;b(x)b(y - h(x)) (13) 
The constants c, ca,/?, K and Ka are nonnegative. The inequalities above determine the 
permissible interaction between the slow and fast variables. They are basically smoothness 
requirements of f and g. 
After these introductory remarks the stability criterion is now stated: 
Theorem: Suppose that conditions 1-3 hold; let d be a positive number such that 
0 < d < 1, and let e*(d) be the positive number given by 
a 7 q- [f/ (1 - d) q- f/2d]2/4d(1 - d) (14) 
where/32 = 1(2 q- C2, 7 - K q- C, then for all e < e*(d), the origin (x = y = O) 
is an asymptotically stable equilibrium point of (3) and (4) and 
v(x, y) - (1 - d)V(x) + dW(x, y) 
(15) 
is a Lyapunov function of (3) and 
1 
If we put e -- X as a global neural time constant in equation (1) then we have 
to determine two Lyapunov functions: one for the boundary-layer system and the 
other for the reduced-order system. 
In [CG83] is mentioned a global Lyapunov function for a competitive neural network 
with only an activation dynamics. 
j,k=l 
(16) 
under the constraints: mij = mji, ai(xi) > 0, h(xj) > O. 
This Lyapunov-function can be t.aken as one for the boundary-layer system (STM- 
equation) , if the LTM contribution Si is considered as a fixed unknown parameter: 
Quadratic-type Lyapunov Functions for Competitive Neural Networks 341 
W(x, S) = E J aj((j )fj ((j)d(j- E Bj Sj a:j f.i ((.i)d(j- E Dij f.i (xj)f(x) 
j=l j=l j=l 
(17) 
For the reduced-order system (LTM-equation) we can take as a Lyapunov-function: 
N 
v(s) = 5ss = s? (x8) 
2 
i--1 
The Lyapunov-function for the coupled STM and LTM dynamics is the sum of the 
two Lyapunov-function: 
v(x, s) = (1 - a)v(s) + aw(x, s) 
(19) 
4 
DESIGN OF STABLE COMPETITIVE NEURAL 
NETWORKS 
Competitive neural networks with learning rules have moving equilibria during the 
learning process. The concept of asymptotic stability derived from matrix pertur- 
bation theory can capture this phenomenon. 
We design in this section a competitive neural network that is able to store a desired 
pattern as a stable equilibrium. 
The theoretical implications are illustrated in an example of a two neuron network. 
Example: Let N = 2, ai = A, Sj -' B, Dii -' o > O, Dij -- -] < 0 and the 
nonlinearity be a linear function .l'(zl) = zj in equations (1) and (2). 
We get for the boundary-layer system: 
N 
i=1 
and for the reduced-order system: 
(20) 
B 1] C 
 = S[A-  A- 
Then we get for the Lyapunov-functions: 
(21) 
and 
I 2 
v(s) = (& + s]) 
(22) 
A 2 A 2 1 
W(x, S) = x + . - BS - S.. - [ + x - 
(23) 
342 A. MEYER-BASE 
0 
-0.2 
-0.4 
-0.6 
-0.8 
-1 
-1.2 
0 1 2 3 4 5 6 7 8 9 10 
time in msec 
Figure 1: Time histories of the neural network with the origin as an equilibrium 
point: STM states. 
For the nonnegative constants we get: rtl = 1 B (A rt) 2, cl 7 = -B, 
A--or  Or2 -- -- -- 
withB<0,andc2=fl=/32=landK=K2=0. 
We get some interesting implications from the above results as: A-ct > B, A-ct > 0 
and B < 0. 
The above impications can be interpreted as follows: To achieve a stable equilibrium 
point (0, 0) we should have a negative contribution of the external stimulus term 
and the sum of the excitatory and inhibitory contribution of the neurons should 
be less than the time constant of a neuron. An evolution of the trajectories of the 
STM and LTM states for a two neuron system is shown in figure 1 and 2. The 
STM states exhibit first an oscillation from the expected equilibrium point, while 
the LTM states reach monotonically the equilibrium point. We can see from the 
pictures that the equilibrium point (0, 0) is reached after 5 msec by the STM- and 
LTM-states. 
Choosing B = -5, A = 1 and ct = 0.5 we obtain for *(d): *(d) = ''5 
55+ 4a  
From the above formula we can see that * (d) has a maximum at d = d* = 0.5. 
5 CONCLUSIONS 
We presented in this paper a quadratic-type Lyapunov function for analyzing the 
stability of equilibrium points of competitive neural networks with fast and slow 
dynamics. This global stability analysis method is interpreting neural networks 
as nonlinear singularly perturbed systems. The equilibrium point is constrained 
to a neighborhood of (0, 0). This technique supposes a monotonically increasing 
non-linearity and a symmetric lateral inhibition matrix. The learning rule is a 
deterministic Hebbian. This method gives an upper bound on the perturbation 
Quadratic-type Lyapunov Functions for Competitive Neural Networks 343 
0 $ 
o 3 
1 2 3 4 5 
time in msec 
0 I I I I 
0 6 7 8 9 10 
Figure 2: Time histories of the neural network with the origin as an equilibrium 
point: LTM states. 
parameter and such an estimation of a maximal positive neural time-constant. The 
practical implication of the theoretical problem is the design of a competitive neural 
network that is able to store a desired pattern as a stable equilibrium. 
References 
[Ama82] 
[Area83] 
[CG83] 
[Gro76] 
[Heb49] 
[SK84] 
S. Amari. Competitive and cooperative aspects in dynamics of neural exci- 
tation and self-organization. Competition and cooperation in neural net- 
works, 20:1-28, 7 1982. 
S. Amari. Field theory of self-organizing neural nets. IEEE Transactions 
on systems, machines and communication, SMC-13:741-748, 7 1983. 
A.M. Cohen und S. Grossberg. Absolute Stability of Global Pattern For- 
mation and Parallel Memory Storage by Competitive Neural Networks. 
IEEE Transactions on Systems, Man and Cybernetics, SMC-13:815-826, 
9 1983. 
S. Grossberg. Adaptive Pattern Classification and Universal Recording. 
Biological Cybernetics, 23:121-134, I 1976. 
D. O. Hebb. The Organization of Behavior. J. Wiley Verlag, 1949. 
All Saberi und Hassan Khalil. Quadratic-Type Lyapunov Functions for 
Singularly Perturbed Systems. IEEE Transactions on Automatic Control, 
pp. 542-550, June 1984. 
