160 Tang 
Analytic Solutions to the Formation of 
Feature-Analysing Cells of a Three-Layer 
Feedforward Visual Information 
Processing Neural Net 
D.S. Tang 
Microelectronics and Computer Technology Corporation 
3500 West Balcones Center Drive 
Austin, TX 78759-6509 
email: tang@mcc.com 
ABSTRACT 
Analytic solutions to the information-theoretic evolution equa- 
tion of the connection strength of a three-layer feedforward neural 
net for visual information processing are presented. The results 
are (1} the receptive fields of the feature-analysing cells corre- 
spond to the eigenvector of the maximum eigenvalue of the Fred- 
holm integral equation of the first kind derived from the evolution 
equation of the connection strength; (2} a symmetry-breaking 
mechanism {parity-violation} has been identified to be respon- 
sible for the changes of the morphology of the receptive field; 
(3} the conditions for the formation of different morphologies are 
explicitly identified. 
1 INTRODUCTION 
The use of Shannon's information theory ( Shannon and Weaver, 1949) to the study 
of neural nets has been shown to be very instructive in explaining the formation 
of different receptive fields in the early visual information processing, as evident by 
the works of Linsker (1986,1988). It has been demonstrated that the connection 
strengths which maximize the information rate from one layer of neurons to the 
next exhibit center-surround, all-excitatory/all-inhibitory and orientation-selective 
properties. This could lead to a better understanding on the mechanisms with 
which the cells are self-organized to achieve adaptive responses to the changing en- 
viroment. However, results from these studies are mainly numerical in nature and 
therefore do n.o,t provide deeper insights as to how and under what conditions the 
morphologies of the feature-analyzing cells are formed. We present in this paper 
Analytic Solutions to the Formation of Feature-Analysing Cells 161 
accurate analytic solutions to the problems posed by Linsker. Namely, we solve 
analytically the evolution equation of the connection strength, obtain close expres- 
sions for the receptive fields and derive the formation conditions for different classes 
of morphologies. These results are crucial to the understanding of the architecture 
of neural net as an information processing system. Below, we briefly summarize the 
analytic techniques involved and the main results we obtained. 
2 THRIIE-IAYIIR $lllID$ORWARD NEURAL NET 
The neural net configuration (Fig. 1) is identical to that reported in references 2 
and 3 in which a feedforword three-layer neural net is considered. The layers are 
labelled consecutively as layer-A, layer-B and layer-C. 
LAYER A 
LA Y' B 
LAY" C 
Figure 1: The neural net configuration 
The input-output relation for the signals to propagate from one layer to the con- 
secutive layer is assumed to be linear, 
My = Z c'y,(L, + 
n is assumed to be an additive Gaussian white noise with constant standard devi- 
ation a and sero mean. i and My are the ith stochastic input signal and the jth 
stochastic output signal respectively. C'y is the connection strength which defines 
the morphology of the receptive teld and is to be determined by maximizing the 
information rate. The spatial summation in equation (1) is to sum over all Ary 
162 Tang 
inputs located according to a gaussian distributed within the same layer, with the 
center of the distribution lying directly above the location of the My output signal. 
If the statistical behavior of the input signal is assumed to be Gaussian, 
P(L) = . , 
(2r) ' x/Dt(Q) 
then the information rate can be derived and is given by 
lln[1 + 
R(4) =   E c" 
(2) 
The matrix Q is the correlation of the L's, Q,i = E[(L,- L)(L i - L)] with mean ,. 
The set of connection strengths which optimize the information rate subject to a 
normalization condition,  C' -- A, and to their overall absolute mean, ( Ci)  - 
B, constitute physically plausible receptive fields. Below is the solutions to the 
problem. 
3 FKEDHOI. M INTEGRAL EQUATION 
The evolution equation for the connection strength C, which maximizes the infor- 
mation rate subject to the constraints is 
N 
1 
 =  (Q, + :)c,. () 
i----1 
k is the Lagrange multiplier. First, we assume that the statistical ensemble of 
the visual images has the highest information content under the condition of fixed 
variance. Then, from the maximum entropy principle, it can be shown that the 
Gaussian distribution with a correlation Qij being a constant multiple of the kro- 
necker delta function describes the statistics of this ensemble of visual images. It 
can be shown that the solution to the above equation with Q,i being a kronecker 
delta function is a constant. Therefore, the connection strengths which defines the 
linear input-output relation from layer A to layer B is either all-excitatory or all- 
inhibitory. Hence, without loss of generality, we take the values of the layer A to 
layer B connection strengths to be all-excitatory. Making use of this result, the 
correlation function of the output signals at layer B (i.e. the input signals to layer 
C) is derived 
Q.. = cc:v(-,.V2,. ) () 
where r is the distance between the nth and the ith output signals. 
50. To study the connection strengths of the input-output relation from layer B to 
layer C, it is more convenient to work with continuous spatial variables. Then the 
solutions to the discrete evolution equation which maximizes the information rate 
are solutions to the following Fredholm integral equation of the first kind with the 
maximum eigenvalue ,, 
C(O=  
N :(IOC() () 
Analytic Solutions to the Formation of Feature-Analysing Cells 163 
where the kernal is K(lr- ) = {Q(-r-)+ka)p(/} and the Gaussian input population 
distribution density is ,o( = Cpcxp{--  } with Cp = N 
,r-' In continuous variables, 
the connection strength is denoted by C{r-). A complete set of solutions to this 
Fredholm integral equation can be analytically derived. We are interested only in 
the solutions with the maximum eigenvalues. Below we present the results. 
ANALYTIC SOLUTIONS 
The solution with the maximum eigenvalue has a few number of nodes. This can 
be constructed as a linear superposition of an infinite number of gaussian functions 
with different variances and means, which are treated as independent variables to be 
solved with the Fredholm integral equation. Full details are contained in reference 
3. 
(a) Symmetric solution C(- = C(: 
For ka  0, the connection strength is 
S 
(1- H) GczP(-2 )] (7) 
with G =  and H = 
' ' 
Liu.= 0.73205 and   
The eigenvalue is given by 
Here, a a = .Sr, o = 0.66667, 
(8) 
For ka = 0, the connection strength is 
C(r = fczp(-2---) 
(9) 
and the eigenvalue is 
= 
f 1 1 ' 
+ 
These can be shown to be identical to the case of 
appropriately taken. 
(b) Antisymmetric solution C(- = 
The connection strength  
.a 
C(r = (fz + gy)zp(- 2- [1- 
1 
1 + , + L,. l). (11) 
The eigenvalue is 
. --- 'CcCt' (12) 
164 Tang 
In the above equations, b, ! and g are normalization constants. 
Below are the conditions under which the different morphologies {Fig.2 } are formed. 
{i}/2 > 0, the symmetric solution has the largest eigenvalue. The receptive field is 
either all-excltatory or all-inhibitory, Fig.2a. 
{i/}-0.891C' < /2 < 0, the symmetric solution has the largest eigenvalue. The 
receptive field has a mexcian-hat appearance, Fig.2b. 
{iii}/a < -0.891C, the anti-symmetric solution has the largest eigenvalue. The 
receptive field has two regions divided by a straight line of arbitrary direc- 
tlon{degeneracy}. The two regions are mirror image of each other. One is totally 
inhibitory and the other is totally excitatory, Fig.2c. 
0.4 
0.35 
0.3 
0.2 
Symmetric solution 
Ant/symmetric solution 
O'05-3 .l .1 o I 1 
Figure 2: Relations between the receptive field and the maximum eigenvalues. 
Inserts are examples of the connection strength C'(O versus the spatial 
dimension in the x-direction. 
Analytic Solutions to the Formation of Feature-Analysing Cells 165 
Note that the formation rate as given by Eq.(3) is invariant under the operation 
of the spatial retlection,-'--, r-'. The solutions to the optimaziation problem violates 
parity-conservation as the overall mean of the connection strength {i.e. equivalently 
/ca} changes to different values. 
Results from numerical simulations agree very well with the analytic results. Nu- 
merical simulations are performed from $0 to 600 synapses. The agreement is good 
even for the case in which the number of synapses are 200. 
In summary, we have shown precisely how the mexican-hat morphology emerges as 
identified by (ii} above. Furthermore, a symmetry-breaking(parity-violation} mech- 
anism has been identified to explain the changes of the morphology from spatially 
symmetric to anti-symmetric appearance as/ca passes through -0.891C'Q. It is very 
likely that similar symmetry breaking mechanisms are present in neural nets with 
lateral connections. 
References
1. C.E.Shannon and W. Weaver, The mathematical Theory of Communication 
{Univ. of Illinois Press,Urbana, 1949}. 
2. R.Linsker, Proc. Natl. Acad. Sci. USA 83,7508{1986}; Computer 21 {3}, 
3. D.S. Tang, Phys. Rev A, 40,6626{1989}. 
