Predicting Complex Behavior 
in Sparse Asymmetric Networks 
All A. Minai and William B. Levy 
Department of Neurosurgery 
Box 420, Health Sciences Center 
University of Virginia 
Charlottesville, VA 22908 
Abstract 
Recurrent networks of threshold elements have been studied inten- 
sively as associative memories and pattern-recognition devices. While 
most research has concentrated on fully-connected symmetric net- 
works, which relax to stable fixed points, asymmetric networks show 
richer dynamical behavior, and can be used as sequence generators or 
flexible pattern-recognition devices. In this paper, we approach the 
problem of predicting the complex global behavior of a class of ran- 
dom asymmetric networks in terms of network parameters. These net- 
works can show fixed-point, cyclical or effectively aperiodic behavior, 
depending on parameter values, and our approach can be used to set 
parameters, as necessary, to obtain a desired complexity of dynamics. 
The approach also provides qualitative insight into why the system 
behaves as it does and suggests possible applications. 
1 INTRODUCTION 
Recurrent neural networks of threshold elements have been intensively investigated in 
recent years, in part because of their interesting dynamics. Most of the interest has 
focused on betworks with symmetric connections, which always relax to stable fixed 
points (Hopfield, 1982) and can be used as associative memories or pattern-recognition 
devices. Networks with asymmetric connections, however, have the potential for much 
556 
Predicting Complex Behavior in Sparse Asymmetric Networks 557 
richer dynamic behavior and may be used for learning sequences (see, e.g., Amari, 1972; 
Sompolinsky and Kanter, 1986). 
In this paper, we introduce an approach for predicting the complex global behavior of an 
interesting class of random sparse asymmetric networks in terms of network parameters. 
This approach can be used to set parameter values, as necessary, to obtain a desired 
activity level and qualitatively different varieties of dynamic behavior. 
2 NETWORK PARAMETERS AND EQUATIONS 
A network consists of n identical 0/1 neurons with threshold 0. The fixed pattern of 
excitatory connectivity between neurons is generated prior to simulation by a Bernoulli 
process with a probability p of connection from neuron j to neuron i. All excitatory 
connections have the fixed value w, and there is a global inhibition that is linear in the 
number of active neurons. If m (t) is the number of active neurons at time t, K the inhi- 
bitory weight, Yi (t) the net excitation and z (t) the firing status of neuron i at t, and cij a 
0/1 variable indicating the presence or absence of a connection from j to i, then the 
equations for i are: 
w  cozj(t-1) 
Yi (t) = I=' , 1 < rn (t-l) < n (1) 
w j cijzj (t-l) + Km (t-l) 
zi(t)=f ify(t) > 0 
otherwise , 0 < 0 < 1 (2) 
If rn (t-l) = 0, Yi (t) = 0 i. Equation (1) is a simple variant of the shunting inhibition 
neuron model studied by several researchers, and the network is similar to the one pro- 
posed by Mart (Mart, 1971). Note that (1) and (2) can be combined to write the neuron 
equations in a more familiar subtractive inhibition format. Defining x = OK/(1-0)w, 
zi(t)={10 if j__ CijZj(t-1) -O__ Zj(t-1) > O (3) 
otherwise 
3 NETWORK BEHAVIOR 
In this paper, we study the evolution of total activity, m (t), as the system relaxes. From 
Equation (3), the firing condition for neuron i at time t, given the activity rn (t-1)=M at 
time t-l, is: el(t) --cijzj(t-1) > 4 r. Thus, in order to fire at time t, neuron i must 
have at least [4] active inputs. This allows us to calculate the average firing probability 
of a neuron given the prior activity M as: 
P{# of active inputs >[M] } = a/tl (kM)pk (1--p)M--p(M;n,p,oO (4) 
k 
If M is large enough, we can use a Gaussian approximation to the 'binomial distribution 
558 Minai and Levy 
and a hyperbolic tangent approximation to the error function to get 
where 
x -- -up 
lMp (1-p ) 
Finally, when M is large enough to assume [o0[/1 = aM, we get an even simpler form: 
where 
(6) 
T - 1 'x cp]'') {x p 
- {x-p  2 ' 
Assuming that neurons fire independently, as they will tend to do in such large, sparse 
networks (Minai and Levy, 1992a,b), the network's activity at time t is distributed as 
P {m(t)=N I m(t-1)=M} = (I) P (Myv (1-@(M)) n-N (7) 
which leads to a stochastic return map for the activity: 
m(t)=n@(m(t-1)) + O ('n) (8) 
In Figure 1, we plot m (t) against m (t-l) for a 120 neuron network and two different 
values of Ix. The vertical bars show two standard deviations on either side of 
n p(m (t-l)). It is clear that the network's activity falls within the range predicted by (8). 
After an initial transient period, the system either switches off permanently (correspond- 
ing to the zero activity fixed point) or gets trapped in an O (4'fi'n) region around the point 
 defined by m (t) = m (t-l). We call this the attracting region of the map. The size and 
location of the attracting region are determined by {x and largely dictate the qualitative 
dynamic behavior of the network. 
As {x ranges from 0 to 1, networks show three kinds of behavior: fixed points, short 
cycles, and effectively aperiodic dynamics. Before describing these behaviors, however, 
we introduce the notion of available neurons. Let ki be the number of input connections 
to neuron i (thefan-in of i). Given m (t-l) = M, if ki < [aM], neuron i cannot possibly 
meet the firing criterion at time t. Such a neuron is said to be disabled by activity M. The 
group of neurons not disabled are considered available neurons. At any specific activity 
M, there is a unique set, No (M), of available neurons in a given network, and only neu- 
rons from this set can be active at the next time step. Clearly, No (M) No (M2) if 
M > M2. The average size of the available set at a given activity M is 
n,(M;n,p,oO--n [1-P{ki ()Pk(1-P) n-k (9) 
ktl 
Predicting Complex Behavior in Sparse Asymmetric Networks 559 
12o 
(a) Effectlvdy Aperiodic Behavior 
0 - 0.85, K - 0.016, w - 0.4 0 ct - 0.227 
o -mW, irki ,, (2O00 q) 
(b) High Activity Cyde 
0 = 0.85, g = 0.012, w =0.4 --------,t = 0.17 
o= ,.--plrk:d  (200 
o toO) 120 0 toO) 120 
Figure 1' Predicted Distribution of rn (t+l) given m (t), and Empirical Data (o) for Two 
Networks A and B. The vertical bars represent 4 standard deviations of the predicted dis- 
tribution for each m (t). Note that the empirical values fall in the predicted range. 
(a) F, ffecfivdy AperiodSc Behavior 
O = 0.85, g = 0.016, w = 0.4  (x = 0.227 
120 
re(t) 
0.4  t =0.17 
(c) Low Activity Cyele 
O - 0.85, ff - 0.0247, w - 0.4 = (x - 0.35 
time step t 400 2,00 b atep t 
10000 
Figure 2: Activity time-series for three kinds of behavior shown by a 120 neuron net- 
work. Graphs (a) and (b) correspond to the data shown in Figure 1. 
:560 Minai and Levy 
It can be shown that no (M) > n p(M), so there are usually enough neurons available to 
achieve the average activity as per (8). 
We now describe the three kinds of dynamic behavior exhibited by our networks. 
(1) Fixed Point Behavior: If e is very small,  is close to n, inhibition is not strong 
enough to control activity and almost all neurons switch on permanently. If e is too 
large,  is close to 0 and the stochastic dynamics eventually finds, and remains at, 
the zero activity fixed point. 
(2) Effectively Aperiodic Behavior: While deterministic, finite state systems such as 
our networks cannot show truly aperiodic or chaotic behavior, the time to repeti- 
tion can be so long as to make the dynamics effectively aperiodic. This occurs 
when the attracting region is at a moderate activity level, well below the ceiling 
defined by the number of available neurons. In such a situation, the network, start- 
ing from an initial condition, successively visits a very large number of different 
states, and the activity, rn (t), yields an effvectively aperiodic time-series of ampli- 
tude O (x/ '), as shown in Figure 2(a). 
(3) Cyclical Behavior: If the attracting region is at a high activity level, most of the 
available neurons must fire at every time step in order to maintain the activity 
predicted by (8). This forces network states to be very similar to each other, which, 
in turn, leads to even more similar successor states and the network settles into a 
relatively short limit cycle of high activity (Figure 2(b)). When the attracting 
region is at an activity level just above switch-off, the network can get into a low- 
activity limit cycle mediated by a very small group of high fan-in neurons (Figure 
2(c)). This effect, however, is unstable with regard to initial conditions and the 
value of e; it is expected to become less significant with increasing network size. 
(a) Mean - 0.287 
Variance  0.0689 
Co) Mean - 0.310 
Variance - 0.0003 
in'wn''ln rrr n n nnn [ 
I I 
O 0.2 FiFii Proimbility 0.75 I O 0.2S 0.75 
Firln Probabmty 
Figure 3: Neuron firing probability histograms for two 120-neuron networks in the effec- 
tively aperiodic phase (e-' 0.227). Graph (a) is for a network with random connectivity 
generated through a Bernoulli process with p = 0.2, while Graph (b) is for a network 
with a fixed fan-in of exactly 24, which corresponds to the mean fan-in for p = 0.2. 
Predicting Complex Behavior in Sparse Asymmetric Networks 561 
One interesting issue that arises in the context of effectively aperiodic behavior is that of 
state-space sampling within the O (]fi') constraint on activity. We assess this by looking 
at the histogram of individual neuron firing rates. Figure 3(a) shows the histogram for a 
120 neuron network in the effectively aperiodic phase. Clearly, some subspaces are being 
sampled much more than others and the histogram is very broad. This is mainly due to 
differences in the fan-in of individual neurons, and will diminish in larger networks. Fig- 
ure 3Co) shows the neuron firing histogram for a 120 neuron network where each neuron 
has a fan-in of 24. The sampling is clearly much more "ergodic" and the dynamics less 
biased towards certain subspaces. 
0.75 -- 
o o y o.7  
Figure 4: The complete set of non-zero activation values available to two identical neu- 
rons i and j with fan-in 24 in a 120-neuron network. 
562 Minai and Levy 
4 ACTIVATION DYNAMICS 
While our modeling so far has focused on neural firing, it is instructive to look at the 
underlying neuron activation values, Yi. If m (t-l)=M, the possible yi(t) values for a 
neuron i with fan-in ki are given by the set 
{ wq ]MAX(O, ki_n+M)<q<MIN(M, ki)I M>O (10) 
Y(M, ki) -= wq + KM 
with Y(0, ki)- = {0}. Here q represents the number of active inputs to i, and the set 
Yi -- sY(M, ki) represents the set of all possible activation values for the neuron. The 
network's n-dimensional activation state, y(t) -- [yl, Y2, ..., Y,, ], evolves upon the activa- 
tion space Y1 x Y2 x  -- x Yn, which is an extremely complex but regular object. In 
Figure 4, we plot a 2-dimensional subspace projection -- called a y-y plot - of the 
activation space for a 120-neuron network excluding the zero states. Both neurons shown 
have a fan-in of 24. In actuality, only a small subset of the activation space is sampled 
due to the constraining effects of the dynamics and the improbability of most q values. 
5 RELATING THE ACTIVITY LEVEL TO (x 
From a practical standpoint, it would be useful to know how the average activity in a net- 
work is related to its x parameter. This can be done using the hyperbolic tangent approxi- 
mation of Equation (6). First, we define the activity level at time t as r (t) -- n-m (t), i.e., 
the proportion of active neurons. This is a macrostate variable in the sense of (Amari, 
1974). In the long term, the activity level becomes confined to a O (1/n) region around 
the value corresponding to the activity fixed point Thus, it is reasonable to use F as an 
estimate for the time-averaged activity level (r). To relate r (and thus F) to x, we must 
solve the fixed point equation  = n p(r). Substituting this and the definition of F into 
0.75 -- 
0 
O 0.02 0.04 
o.1 
Figure 5: Predicted and empirical activities for 1000 neuron networks with p = 0.05. 
Each data point is averaged over 7 networks. 
Predicting Complex Behavior in Sparse Asymmetric Networks 563 
(6) gives: 
tz(') =p +  p (l-p) tanh_(1 _ 2' ) (11) 
2n 
While c can range from 0 to 1, the approximation of (11) breaks down at very high or 
very small values of 7. However, the range of its applicability gets wider as n increases. 
Figure 5 shows the performance of (11) in predicting the average activity level in a 
1000-neuron network. Note that c =p always leads to  = 0.5 by Equation (11). 
6 CONCLUSION 
We have studied a general class of asymmetric networks and have developed a statistical 
model to relate its dynamical behavior to its parameters. This behavior, which is largely 
characterized by a composite parameter a, is richly varied. Understanding such behavior 
provides insight into the complex possibilities offered by sparse asymmetric networks, 
especially with regard to modeling such brain regions as the hippocampal CA3 area in 
mammals. The complex behavior of random asymmetric networks has been discussed 
before by Parisi (Parisi, 1986), Ntitzel (Niitzel, 1991), and others. We show how to con- 
trol this complexity in our networks by setting parameters appropriately. 
Acknowledgements: This research was supported by NIMH MH00622 and NIMH 
MH48161 to WBL, and by the Department of Neurosurgery, University of Virginia, Dr. 
John A. Jane, Chairman. 
References 
S. Amari (1972). Learning Patterns and Pattern Sequences by Self-Organizing Nets of 
Threshold Elements. IEEE Trans. on Computers C-21, 1197-1206 
S. Amari (1974). A Method of Statistical Neurodynamics. Kybernetik 14, 201-215 
J.J. Hopfield (1982). Neural Networks and Physical Systems with Emergent Collective 
Computational Abilities. Proc. Nat. Acad. Sci. USA 79, 2554-2558. 
D. Mart (1971). Simple Memory: A Theory for Archicortex. Phil. Trans. R. Soc. Lond. 
B 262, 23-81. 
A.A. Minai and W.B. Levy (1992a). The Dynamics of Sparse Random Networks. In 
Review. 
A.A. Minai and W.B. Levy (1992b). Setting the Activity Level in Sparse Random Net- 
works. In Review. 
K. Niitzel (1991). The Length of Attractors in Asymmetric Random Neural Networks 
with Deterministic Dynamics. J. Phys. A: Math. Gen 24, L151-157. 
G. Parisi (1982). Asymmetric Neural Networks and the Process of Learning. J. Phys. A: 
Math. Gen. 19, L675-L680. 
H. Sompolinsky and I. Kanter (1986), Temporal Association in Asymmetric Neural Net- 
works. Phys. Rev. Lett. 57, 2861-2864. 
