74O 
SPATIAL ORGANIZATION OF NEURAL NETWORKS: 
A PROBABILISTIC MODELING APPROACH 
A. Stafylopati s 
M. Dikaiakos 
D. Kontoravdi s 
National Technical University of Athens, Department of Electri- 
cal Engineering, Computer Science Division, 157 73 Zographos, 
Athens, Greece. 
ABSTRACT 
The aim of this paper is to explore the spatial organization of 
neural networks under Markovian assumptions, in what concerns the be- 
haviour of individual cells and the interconnection mechanism. Space- 
organizational properties of neural nets are very relevant in image 
modeling and pattern analysis, where spatial computations on stocha- 
stic two-dimensional image fields are involved. As a first approach 
we develop a random neural network model, based upon simple probabi- 
listic assumptions, whose organization is studied by means of dis- 
crete-event simulation. We then investigate the possibility of ap- 
proximating the random network's behaviour by using an analytical ap- 
proach originating from the theory of general product-form queueing 
networks. The neural network is described by an open network of no- 
des, in which customers moving from node to node represent stimula- 
tions and connections between nodes are expressed in terms of sui- 
tably selected routing probabilities. We obtain the solution of the 
model under different disciplines affecting the time spent by a sti- 
mulation at each node visited. Results concerning the distribution 
of excitation in the network as a function of network topology and 
external stimulation arrival pattern are compared with measures ob- 
tained from the simulation and validate the approach followed. 
INTRODUCTION 
Neural net models have been studied for many years in an attempt 
to achieve brain-like performance in computing systems. These models 
are composed of a large number of interconnected computational ele- 
ments and their structure reflects our present understanding of the 
organizing principles of biological nervous systems. In the begin- 
ing, neural nets, or other equivalent models, were rather intended 
to represent the logic arising in certain situations than to provide 
an accurate description in a realistic context. However, in the last 
decade or so the knowledge of what goes on in the brain has increased 
tremendously. New discoveries in natural systems, make it now rea- 
sonable to examine the possibilities of using modern technology in 
order to synthesige sys.ems that have some of the properties of real 
neural systems 8,9,10,1 
In the original neural net model developed in 1943 by McCulloch 
and Pitts 1,2 the network is made of many interacting components, 
known as the "McCulloch-Pitts cells" or "formal neurons", which are 
simple logical units with two possible states changing state accord- 
American Institute of Physics 1988 
741 
ing to a threshold function of their inputs. Related automata moel. 
have been used later for gene control systems (genetic networks) , 
in which genes are represented as binary automata changing state ac- 
cording to boolean functions of their inputs. Boolean networks con- 
stitute a more general model, whose dynamical behaviour has been stu- 
died extensively. Due to the large number of elements, the exact 
structure of the connections and the functions of individual compo- 
nents are generally unknown and assumed to be distributed at random. 
Several studies on these random boolean networks 5,6 have shown that 
they exhibit a surprisingly stable behaviour in what concerns their 
temporal and spatial organization. However, very few formal analyti- 
cal results are available, since most studies concern statistical 
descriptions and computer simulations. 
The temporal and spatial organization of random boolean networks 
is of particular interest in the attempt of understanding the proper- 
ties of such syste.ms, and models originating from the theory of sto- 
chastic processes l seem to be very useful. Spatial properties of 
neural nets are most important in the field of image recognition 12. 
In the biological eye, a level-normalization computation is pe.rformed 
by the layer of horizontal cells, which are fed by the immediately 
preceding layer of hotoreceptors. The horizontal cells take the 
outputs of the receptors and average them spatial ly, this average 
being weighted on a nearest-neighbor basis. This procedure corres- 
ponds to a mechanism for determining the brightness level of pixels 
in an image field by using an array of processing elements. The 
principle of local computation is usually adopted in models used for 
representing and generating textured images. Among the stochastic 
models applied to analyzi the parameters of image fields, the ran- 
dom Markov field model /, seems to give a suitably structured re- 
presentation, which is mainly due to the application of the marko- 
vian property in space. This type of modeling constitutes a promi- 
sing alternative in the study of spatial organization phenomena in 
neural nets. 
The approach taken in this paper aims to investigate some as- 
pects of spatial organization under simple stochastic assumptions. 
In the next section we develop a model for random neural networks 
assuming boolean operation of individual cells. The behaviour of 
this model, obtained through simulation experiments, is then appro- 
ximated by using techniques from the theory of queueing networks. 
The approximation yields quite interesting results as illustrated by 
various examples. 
THE RANDOM NETWORK MODEL 
We define a random neural network as a set of elements or cells, 
each one of which can be in one of two different states: firing or 
quiet. Cells are interconnected to form an NxN grid, where each grid 
point is occupied by a cell. We shall consider only connections be- 
tween neighbors, so that each cell is connected to 4 among the other 
cells- two input and two output cells (the output of a cell is equal 
'to its internal state and it is sent to its output cells which use 
.it as one of their inputs). The network topology is thus specified 
742 
by its incidence matrix A of dimension MxM, where M=N 2. This matrix 
takes a simple form in the case of neighbor-connection considered 
here. We further assume a periodic structure of connections in what 
concerns inputs and outputs; we will be interested in the following 
two types of networks depending upon the period of reproduction for 
elementary square modules 5, as shown in Fig.l: 
- Propagative nets (Period 1) 
- Looping nets (Period 2) 
(a) 
(b) m  
Fig.1. (a) Propagative connections, (b) Looping connections 
At the edges of the grid, circular connections are established (mo- 
dulo N), so that the network can be viewed as supported by a torus. 
The operation of the network is non-autonomous: changes of sta- 
te are determined by both the interaction among cells and the influ- 
ence of external stimulations. We assume that stimulations arrive 
from the outside world according to a Poisson process with parameter 
A. Each arriving stimulation is associated with exactly one cell of 
the network; the cell concerned is determined by means of a given 
discrete probability distribution qi (l<_i<_M), considering an one-di- 
mensional labeling of the M cells. 
The operation of each individual cell is asynchronous and can be 
described in terms of the following rules: 
- A quiet cell moves to the firing state if it receives an arriving 
stimulation or if a boolean function of its inputs becomes true. 
- A firing cell moves to the quiet state if a boolean function of its 
inputs becomes false. 
- Changes of state imply a reaction delay of the cell concerned; the- 
se delays are independent identically distributed random variables 
following a negative exponential distribution with parameter V. 
According to these rules, the operation of a cell can be viewed as il- 
lustrated by Fig.2, where the horizontal axis represents time and th 
numbers 0,1,2 and 3 represent phases of an operation cycle. Phases 1 
and 3, as indicated in Fig.2, correspond to reaction delays. In this 
sense, the quiet and firing states, as defined in the beginingofthis 
section, represent the aggregates of phases 0,1 and 2,3 respectively. 
External stimulations affect the receiving cell only when itisinpha- 
se O; otherwise we consider that the stimulation is lost. In the sa- 
me way, we assume that changes of the value of the input boolean func- 
.ion do not affect the operation of the cell during phases I and 3. The 
conditions are checked onlyat the end of the respective reaction delay. 
743 
quiet 
s ta te 
firing state 
0 
Fig.2. Changes of state for individual cells 
The above defined model includes some features of the original 
McCulloch-Pitts cells 1,2 . In fact, itrepresents an asynchronous 
counterpart of the latter, in which boolean functions are considered 
instead of threshold functions. However, it can be shown that any 
McCulloch and Pitts' neural network can be implemented by a boolean 
network designed in an appropriate fashion 5. In what follows,wewill 
consider that the firing condition for each individual cell is de- 
termined by an "or" function of its inputs. 
Under the assumptions adopted, the evolution of the network in 
time can be described by a continuous-parameter Markov process. How- 
ever, the size of the state-space and the complexity of the system 
are such that no analytical solution is tractable. The spatial orga- 
nization of the network could be expressed in terms of the steady- 
state probability distribution for the Markov process. A more useful 
representation is provided by the marginal probability distributions 
for all cells in the network, or equivalently by the probability of 
being in the firing state for each cell. This measure expresses the 
level of excitation for each point in the grid. 
We have studied the behaviour of the above model by means of si- 
mulation experiments for various cases depending upon the network si- 
ze, the connection type, the distribution of external stimulation ar- 
rivals on the grid and the parameters A and V. Some examples are il- 
lustrated in the last section, in comparison with results obtained 
using the approach discussed in the next section. The estimations ob- 
tained concern the probability of being in the firing state for all 
cells in the network. The simulation was implemented according to 
the "batched means" method; each run was carried out until tliewidth 
of the 95% confidence interval was less that 10% of the estimated 
mean value for each cell, or until a maximum number of events had 
been simulated depending upon the size of the network. 
THE ANALYTICAL APPROACH 
The neural network model considered in the previous section exhi- 
bited the markovian property in both time and space. Markovianity in 
space, expressed by the principle of "neighbor-connections", is the 
basic feature of Markov random fields 7,14, as already discussed. Our 
idea is to attempt an approximation of the random neural network mo- 
del by using a well-known model, which is markovian in time, and ap- 
plying the constraint of markovianity in space. The model considered 
is an open queueing network, which belongs to the general class of 
queueing networks admitting a product-form solution 4. In fact, one 
could distinguish several comm6n features in the two network models. 
744 
A neural network, in general, receives information in the form of ex- 
ternal stimulation signals and performs some computation on this in- 
formation, which is represented by changes of its state. The opera- 
tion of the network can be viewed as a flow of excitement among the 
cells and the spatial distribution of this excitement represents the 
response of the network to the information received. This kindof ope- 
ration is particularly relevant in the processing of image fields. On 
the other hand, in queueing networks, composed of a number of service 
station nodes, customers arrive from the outside world and spend some 
time in the network, during which they more from node to node, wait- 
ing and receiving service at each node visited. Following the exter- 
nal arrival pattern, the interconnection of nodes and the other net- 
work parameters, the operation of the network is characterized by a 
distribution of activity among the nodes. 
Let us now consider a queueing network, where nodes represent 
cells and customers represent stimulations moving from cell to cell 
following the topology of the network. Our aim is to define the net- 
work's characteristics in a way to match those of the neural net mo- 
del as much as possible. Our queueing network model is completely 
specified by the following assumRtions: 
- The network is composed of M=N L nodes arranged on an NxN rectangu- 
lar grid, as in the previous case. Interconnections are expressed by 
means of a matrix R of routing probabilities: rij !l<i,j<M) repre- 
sents the probability that a stimulation (cuStomer} TeavTng node i 
will next visit node j. Since it is anopen network, after visiting an 
arbitrary number of cells, stimulations may eventually leave the net- 
work. Let .rio denote the probability of leaving the network upon lea- 
ving node 1. In What follows, we will assume that rio:s for all nodes. 
In what concerns the routing proba6ilities ri. i, they are determined 
by the two interconnection schemata considered in the previous sec- 
tion (propagative and looping connections): each node i has two out- 
put nodes j, for which the routing probabilities are equally distri- 
buted. Thus, rij:(1-s)/2 for the two output nodes of i and equal to 
zero for all other nodes in the network. 
- External stimulation arrivals follow a Poisson process with parame- 
ter A and are routed to the nodes according to the probability dis- 
tribution qi (l<_i<_M) as in the previous section. 
- Stimulations receive a "service time" at each node visited. Service 
times are independent identically distributed random variables, which 
are exponentially distributed with parameter V. The time spent by a 
stimulation at a node depends also upon the "service discipline" 
adopted. We shall consider two types of service disciplines according 
to the general queueing network model 4: the first-come-first-served 
(FCFS) discipline, where customers are served in the order of their 
arrival to the node, and the infinite-server (IS) discipline, where 
a customer's service is immediately assumed upon arrival to the node, 
as if there were always a server available for each arriving custo- 
mer (the second type includes no waiting delay). We will refer to the 
above two types of nodes as type i and type 2 respectively. In either 
case, all nodes of the network will be of the same type. 
The steady-state solution of the above network is a straight- 
forward application of the general BCMP theorem 4 according to the 
745 
simple assumptions considered. The state of the system is described 
.by the vector (kl,k2,...,kM), where ki is the number of customers 
present t node i. We first define the traffic intensity Pi for each 
node i as 
Pi = Aei/v , i : 1,2,...,M (1) 
where the quantities {ei} are the solution of the following set of 
linear equations: 
M 
ei = qi +  e.r i = 1,2, ,M (2) 
j=l j ji ' '" 
It can be easily seen that, in fact, e i represents the average num- 
ber of visits a customer makes to node i during his sojourn in the 
network. The existence of a steady-state distribution for the system 
depends on the solution of the above set. Following the general theo- 
rem 4, the joint steady-state distribution takes the form of a pro- 
duct of independent distributions for the nodes: 
Where 
p(kl,k2,...,kM) : l(kl)P2(k2)...PM(kM) 
(3) 
k i 
(1-Pi)p i (Type 1) 
Pi(ki ) : k i (4) 
-Pi Pi 
e k---. (Type 2) 
provided that the stability condition Pi<l is satisfied for type 1 
nodes. 
The product form solution of this type of network expresses the 
idea of global and local balance which is characteristic of ergodic 
Markov processes. We can then proceed to deriving the desired measure 
fOP each node in the network; we are interested in the probability of 
being active for each node, which can be interpreted as the probabi- 
lity that at least one customer is present at the node: 
I Pi (Type 1) 
P(ki>O):l-Pi(O) : (5} 
1-e -i (Type 2} 
The variation in space of the above quantity will be studied with res- 
pect to the corresponding measure obtained from simulation experi- 
ments for the neural network model. 
NUMERICAL AND SIMULATION EXAMPLES 
Simulations and numerical solutions of the queueing network mo- 
el were run for different values of the parameters. The network si- 
zes considered are relatively small but can provide useful informa- 
tion on the patial organization of the networks. For both types of 
service discipline discussed in the previous section, the approach 
followed yields a very good approximation of the network's organiza- 
tion in most regions of the rectangular grid. The choice of the pro- 
bability s of leaving the network plays a critical role in the beha- 
746 
i I I I 
I11 
(a) i I I X (b) 
I1 , ><2 >4 
><Y,, x ./ 
I I 
I I _ 
I xxxx x 
Fig.3. A 10x10 network with A=I, V=I and propagative connections. 
External stimulations are uniformly distributed over a 3x3 square 
on the upper left corner of the grid. (a) simulation (b) Queueing 
network approach with s=0.05 and type 2 nodes. 
(a) (b) 
Fig.4. The network of Fig.3 with A=2 (a) Simulation (b) Queueing 
network approach with s=0.08 and type 2 nodes. 
viour of the queueing model,and must have a non-zero value in order 
for the network to be stable. Good results are obtained for very 
small values of s; in fact, this parameter represents the phenomenon 
of excitation being "lost" somewhere in the network. Graphical re- 
presentations for various cases are shown in Figures 3-7. We have 
used a coloring of five "grey levels", defined by dividing into five 
segments the interval between the smallest and the largest value of 
the probability on the grid; the normalization is performed with res- 
pect to simulation results. This type of representation is less ac- 
curate than directly providing numerical values, but is more clear 
for describing the organization of the system. In each case, the 
results shown for the queueing model concern only one type of nodes, 
the one that best fits the simulation results, which is type 2 in 
the majority of cases examined. The graphical representation illu- 
strates the structuring of the distribution of excitation on the 
grid in terms of functionally connected regions of high and low 
747 
(a) 
(b) 
Fig.5. A 10x10 network with A:I, V:I and looping connections. 
External stimulations are uniformly distributed over a 4x4 
square on the center of the grid. (a) Simulation (b) Queueing 
network approach with s=0.07 and type 2 nodes. 
(a) (b) 
I! 
Fig.6. The network of Fig.5 with h=0.5 (a) Simulation (b) Queue- 
ing network approach with s=0.03 and type 2 nodes. 
excitation. We notice that clustering of nodes mainly follows the 
spatial distribution of external stimulations and is more sharply 
structured in the case of looping connections. 
CONCLUSION 
We have developed in this paper a simple continuous-time pro- 
babilistic model of neural nets in an attempt to investigate their 
spatial organization. The model incorporates some of the main fea- 
tures of the McCulloch-Pitts "formal neurons" model and assumes boo- 
lean operation of the elementary cells. The steady-state behaviour 
of the model was approximated by means of a queueing network model 
with suitably chosen parameters. Results obtained from the solution 
of the above approximation were compared with simulation results of 
the initial model, which validate the approximation. This simpli- 
fied approach is a first step in an attempt to study the organiza- 
748 
I I I 
I I I 
IFdX 
x 
(b) 
Fig.7. A 16x16 network with A=I, V=I and looping connections. 
External stimulations are uniformly distributed over two 4x4 
squares on the upper left and lower right corners of the grid. 
(a) Simulation (b) Queueing network approach with s:0.05 and 
type I nodes. 
74q 
tional properties of neural nets by means of markovian modeling te- 
chniques. 
REFERENCES 
1. W. S. McCulloch, W. Pitts, "A Logical Calculus of the Ideas Im- 
manent in Nervous Activity", Bull. of Math. Biophysics 5, 115- 
133 (1943). 
2. M. L. Minky, Computation: Finite and Infinite Machines (Pren- 
tice Hall, 1967). 
3. S. Kauffman, "Behaviour of Randomly Constructed Genetic Nets", 
in Towards a Theoretical Biology, Ed. C. H. Waddington (Edin- 
burgh University Press, 1970). 
4. F. Baskett, K. M. Chandy, R. R. Muntz, F. G. Palacios, "Open, 
Closed and Mixed Networks of Queues with Different Classes of 
Customers", J. ACM, 22 (1975). 
5. H. Atlan, F. Fogelman-Souli, J. Salomon, G. Weisbuch, "Random 
Boolean Networks", Cyb. and Syst. 12 (1981). 
6. F. Folgeman-Souli, E. Goles-Chacc, G. Weisbuch, "Specific Roles 
of the Different Boolean Mappings in Random Networks", Bull. of 
Math. Biology, Vol.44, No 5 (1982). 
7. G. R. Cross, A. K. Jain, "Markov Random Field Texture Models", 
IEEE Trans. on PAMI, Vol. PAMI-5, No I (1983). 
8. E. R. Kandel, J. H. Schwartz, Principles of Neural Science, 
(Elsevier, N.Y., 1985). 
9. J. J. Hopfield, D. W. Tank, "Computing with Neural Circuits: 
A Model", Science, Vol. 233, 625-633 (1986). 
10. Y. S. Abu-Mostafa, D. Psaltis, "Optical Neural Computers", 
Scient. Amer., 256, 88-95 (1987). 
11. R. P. Lippmann, "An Introduction to Computing with Neural Nets", 
IEEE ASSP Mag. (Apr. 1987). 
12. C. A. Mead, "Neural Hardware for Vision", Eng. and Scie. (June 
1987). 
13. E. Gelenbe, A. Stafylopatis, "Temporal Behaviour of Neural Net- 
works", IEEE First Intern. Conf. on Neural Networks, San Diego, 
CA (June 1987). 
14. L. Onural, "A Systematic Procedure to Generate Connected Binary 
Fractal Patterns with Resolution-varying Texture", Sec. Intern. 
Sympt. on Comp. and Inform. Sciences, Istanbul, Turkey (Oct. 
1987). 
