459 
A BIFURCATION THEORY APPROACH TO THE 
PROGRAMMING OF PERIODIC ATTRACTORS IN 
NETWORK MODELS OF OLFACTORY CORTEX 
Bill Baird 
Department of Biophysics 
U.C. Berkeley 
ABSTRACT 
A new learning algorithm for the storage of static 
and periodic attractors in biologically inspired 
recurrent analog neural networks is introduced. 
For a network of n nodes, n stati, c or n/2 periodic 
attractors may be stored. The aigorithm ailows 
programming of the network vector fieid indepen- 
dent of the patterns to be stored. Stability of 
patterns, basin geometry, and rates of convergence 
may be controlled. For orthonormal patterns, the 
legming operation reduces to a kind of periodic 
outer product ruie that allows iocal, additive, 
commutative, incremental learning. Standing or 
traveiing wave cycles may be stored to mimic the 
kind of osciilating spatial patterns that appear 
in the neural activity of the olfactory bulb and 
prepyriform cortex during inspiration and suffice, 
in the buib, to predict the pattern recognition 
behavior of rabbits in ciassical conditioning ex- 
periments. These attractors arise, during simuIat- 
ed inspiration, through a muItipIe Hopf bifurca- 
tion, which can act as a criticaI "decision point" 
for their selection by a very smaIi input pattern. 
INTRODUCTION 
This approach allows the construction of biological models and 
the expioration of engineering or cognitive networks that 
empioy the type of dynamics found in the brain. Patterns of 40 
to 80 hz osciiiation have been observed in the iarge scaie ac- 
tivity of the oifactory bulb and cortex(Freeman and Baird 86) 
and even visuai neocortex(Freeman 87,Grey and Singer 88), and 
found to predict the oifactory and visual pattern recognition 
responses of a trained animai. Here we use anaiytic methods of 
bifurcation theory to design aigorithms for determining synap- 
tic weights in recurrent network architectures, iike those 
460 Baird 
found in olfactory cortex, for associative memory storage of 
these kinds of dynamic patterns. 
The "projection algorithm n introduced here employs higher 
order correlations, and is the most analytically transparent 
of the algorithms to come from the bifurcation theory ap- 
proach(Baird 88). Alternative numerical algorithms employing 
unused capacity or hidden units instead of higher order corr- 
elations are discussed in (Baird 89). All of these methods 
provide solutions to the problem of storing exact analog at- 
tractors, static or dynamic, in recurrent neural networks, and 
allow programming of the ambient vector field independent of 
the patterns to be stored. The stability of cycles or equi- 
libria, geometry of basins of attraction, rates of convergence 
to attractors, and the location in parameter space of primary 
and secondary bifurcations can be programmed in a prototype 
vector field - the normal form. 
To store cycles by the projection algorithm, we start with the 
amplitude equations of a polar coordinate normal form, with 
coupling coefficients chosen to give stable fixed points on 
the axes, and transform to Cartesian coordinates. The axes of 
this system of nonlinear ordinary differential equations are 
then linearly transformed into desired spatial or spario-tem- 
poral patterns by projecting the system into network coordina- 
tes - the standard basis - using the desired vectors as colum- 
ns of the transformation matrix. This method of network syn- 
thesis is roughly the inverse of the usual procedure in bifur- 
cation theory for analysis of a given physical system. 
Proper choice of normal form couplings will ensure that the 
axis attractors are the only attractors in the system - there 
are no nspurious attractors . If symmetric normal form coef- 
ficients are chosen, then the normal form becomes a gradient 
vector field. It is exactly the gradient of an explicit poten- 
tial function which is therefore a strict Liapunov function 
for the system. Identical normal form coefficients make the 
normal form vector field equivariant under permutation of the 
axes, which forces identical scale and rotation invariant 
basins of attraction bounded by hyperplanes. Very complex 
periodic attractors may be established by a kind of Fourier 
synthesis as linear combinations of the simple cycles chosen 
for a subset of the axes, when those are programmed to be 
unstable, and a single 'mixed mode" in the interior of that 
subspace is made stable. Proofs and details on vectorfield 
programming appear in (Baird 89). 
In the general case, the network resulting from the projection 
A Bifurcation Theory Approach to Programming 461 
algorithm has fourth order correlations, but the use of restr- 
ictions on the detail of vector field programming and the 
types of patterns to be stored result in network architectures 
requiring only second order correlations. For biological mod- 
eling, where possibly the patterns to be stored ore sparse and 
nearly orthogonal, the learning rule for periodic patterns 
becomes o "periodic" outer product rule which is local, add- 
itive, commutative, and incremental. It reduces to the usual 
Hebb-like rule for static attractors. 
CYCLES 
The observed physiological activity may be idealized mathe- 
maticalIy as a "ycie", r xj e i(oj +wt) , j=l,2,...,n. Such o 
cycie is o periodic attractor" if zt is stobie. The gioboi 
ompiitude r is just o scoiing factor for the pattern x , and 
the giobol phase w in e iwt is o periodic scaling that scales x 
by o factor between + I at frequency w os t varies. 
The same vector s or "pattern" of relative amplitudes can 
appear in space os o standing wave, like that seen in the 
bulb, if the relative phase 0st of each compartment (component) 
is the same, 0si+ 1 = s i, or os o traveling wave, like that seen 
in the prepyriform cortex, if the relative phase components of 
9 s form o gradient in space, si+ 1 = 1/a s i. The troveIing wave 
will "sweep out" the amplitude pattern s in time, but the 
root-mean-square amplitude measured in on experiment will be 
x s regardless of the phase pattern For on arbitrary 
the same _ , . 
phase vector these "simple" single frequency cycles con make 
very complicated looking spario-temporal patterns. From the 
mathematical point of view, the relative phase pattern 9 is o 
degree of freedom in the kind patterns that con be stored. 
Patterns of uniform amplitude  which differed only in the 
phase locking pattern 9 could be stored os well. 
To store the kind of patterns seen in bulb, the ompIitude 
vector  is assumed to be parsed into equai numbers of excita- 
tory and inhibitory components, where each ciass of component 
has identical phase, but there is o phase difference of 80 - 
g0 degrees between the ciosses. The traveling wove in the 
prepyriform cortex is modeied by introducing on odditionoi 
phase godient into both excitotory and inhibitory ciosses. 
PROJECTION ALGORITHM 
The central result of this paper is most compactly stated as 
the following: 
462 Baird 
THEOREM 
Any set S, s = 1,2, ...,n/2 , of cycles r s xjS ei(ejs+ t) of 
linearly independent vectors of relative component amplitudes 
s ( R n and phases 9 s ( S n, with frequencies w s ( R and global 
amplitudes r s  R, may be established in the vector field of 
the analo fourth order network: 
x i : -  x i + Ej Tij xj + Ejk i Tijki xj x k x i + b i (t) 
by some variant of the projection operation : 
= p-1 
Tij Emn Pim ]rnn nj ' 
= p-1 p-1 
Tijkl Emn Pim Amn mj P-lnk nl ' 
where the n x n matrix P contains the real and imaginary com- 
ponents [s cos s , s sin s] of the complex eigenvectors 
x s e ies as columns, J is an n x n matrix of complex conjugate 
eigenvalues in diagonal blocks, Amn is an n x n matrix of 2x2 
blocks of repeated coefficients of the normal form equations, 
and the input b i (t) is a delta function in time that establ- 
ishes an initial condition. The vector field of the dynamics 
of the global amplitudes r s and phases s is then given exactly 
by the normal form equations : 
2 
r s = u s r s - r s Ej asj rj 
s ' ws + Ej bsj rj 2 
In particular, for ask > 0 , and ass/aks < I , for all s and k, 
the cycles s = 1,2,...,n/2 ore stable, and have amplitudes 
r s = (us/ass) 1/2, where u s = I -  
Note that there is o multiple Hopf bifurcation of codimension 
n/2 at  = 1. Since there ore no approximations here, however, 
the theorem is not restricted to the neighborhood of this 
bifurcotion, and can be discussed without further reference to 
bifurcotion theory The normal form equations for drS/dt and 
dS/dt determine how r s and s for pattern s evolve in time in 
interaction with all the other patterns of the set S. This 
could be thought of os the process of phase locking of the 
pattern that finally emerges The unusual power of this al- 
gorithm lies in the ability to precisely specify these non- 
linear interactions. In general, determination of the modes of 
the linearized system alone (Li and Hopfield 89) is insuf- 
ficient to say what the attractors of the nonlinear system 
will be. 
A Bifurcation Theory Approach to Programming 463 
PROOF 
The proof of the theorem is instructive since it is a constru- 
ctive proof, and we con use it to explain the learning algori- 
thm. We proceed by showing first that there ore always fixed 
points on the axes of these amplitude equations, whose stabil- 
ity is given by the coefficients of the nonlinear terms. /hen 
the network above is constructed from these equations by two 
coordinate transformations. /he first is from polar to Car- 
tesian coordinates, and the second is o linear transformation 
from these canonical "mode" coordinates into the standard 
basis e 1, e 2, ..., e N, or "network coordinates". /his second 
transformation constitutes the "learning algorithm", because 
it trofdrms the simple fixed points of the amplitude equa- 
tions into the specific spario-temporal memory patterns desi- 
red for the network. 
Amplitude Fixed Points 
Because the amplitude equations are independent of the rota- 
tion , the fixed points of the amplitude equations charact- 
erize the asymptotic states of the underlying oscillatory 
modes. The stability of these cycles is therefore given by the 
stability of the fixed points of the amplitude equations. On 
each axis r s , the other components rj ore zero, by definition, 
rj = rj ( uj - 
rs  rs ( Us - 
E k Ojk rk 2 ) = 0 , 
for rj = 0 , 
which leaves 
ass rs 2 ) , and r s = 0 , when rs 2 = us/ass . 
There is an equilibrium on each axis s, at rs=(us/oss )1/2, as 
claimed. Now the Jacobion of the amplitude equations at some 
fixed point r ^ has elements 
ji Oij J ' 
Jij = - 20ij rAi rj , Jli = ui - 5 oii r^i 2 - r ^ 2 
For o fixed point r s on axis s, Jij = 0 , since r i or r^j = 0, 
making J o diagonal matrix whose entries ore therefore its 
eigenvalues. Now 1i = ui - als r s 2, for i /= s, and Jss = us - 
5 ass r s2. Since r s 2 = us/ass, Jss = - 2 u s , and ii = ui - ais 
(Us/ass). This gives Ois/Oss > ui/u s as the condition for nega- 
tive eigenvolues that assures the stability of rs. Choice of 
oD/oii > uj/u i, for oll i,j , therefore guarantees stability of 
oll axis fixed points. 
Coordinate Transformations 
We now construct the neural network from these well behaved 
equations by the following transformations, 
First; polar to Cartesian , (rs,s) to (V2s_l,V2s) : Using 
v2s_ 1 = r s cos m s , V2s : r s sin s ,ond differentioting these 
464 Baird 
gives: 
V2s_ 1 = r s cos e s - r s sin s s , 
V2s = r s sin s + r s cos s % , 
by the chain rule. Now substituting 
and r s sin e s = V2s ' 
gives: 
COS s = V2s-1/rs , 
V2s-1 = (V2s-1/rs) rs - V2s s 
V2s = V2s r s + (V2s_l/r s) e s 
Entering 
gives: 
the expressions of the normal form for r s and s, 
V2s_ 1 = (vzs_l/rs) (U s r s + r s Ej osj rj2) 
rs 2 2 2 
= V2s_ 1 + V2s 
and since 
- v2s (w s + Ej bsj rj2), 
V2s_ 1 = U s V2s-1 - 
n/2 
w s V2s + Ej [V2s_ 1 asj 
- V2s bsj] (Vzj-12 + v2j 2) 
Similarly, 
V2s = U s V2s + W s V2s- 1 
n/2 
v 2 
+ Ej [V2s asj + v2s_ 1 bsj] ( 2J-1 + v2j 2)' 
Setting the bsj = 0 for simplicity, choosing u s = -  + 1 
to get a standard network form, and reindexing i,j=l,2,...,n , 
we get the Cartesian equivalent of the polar normal form equa- 
tions 
 n  
v = -  v i + Ej JiJ vj + v i Ej Aij vj 2 
Here J is a matrix containing 2x2 blocks along the diagonal of 
the local couplings of the linear terms of each pair of the 
previous equations v2s_ 1 , v2s , with -  separated out of the 
diagonal terms. The matrix A has 2x2 blocks of identical coef- 
ficients asj of the nonlinear terms from each pair. 
I - w 1 
w 1 I 
I - W 2 
w 2 I 
Oll Oll o12 o12 
Oll Oll o12 o12 
Q21 Q21 Q22 Q22 
Q21 a21 Q22 Q22 
A Bifurcation Theory Approach to Programming 465 
Learning Transformation - Linear Term 
Second; J is the canonical form of a real matrix with complex 
conjugate eigenvalues, where the conjugate pairs appear in 
blocks along the diagonal as shown. The Cartesian normal form 
equations describe the interaction of these linearly uncoupled 
complex modes due to the coupling of the nonlinear terms. We 
can interpret the normal form equations as network equations 
in eigenvector (or "memory") coordinates, given by some diag- 
onalizing transformation P, containing those eigenvectors as 
its columns, so that J = p-1 T P. Then it is clear that T may 
instead be determined by the reverse projection T = P 
back into network coordinates, if we start with desired eigen- 
vectors and eigenvalues. We are free to choose as columns in 
P, the real and imaginary vectors Ix s cos e s , x s sin e s ] of the 
cycles s eieS of any linearly indeendent-set-S of ptterns 
to be learned. If we write the matrix expression for the proj- 
ection in component form, we recover the expression given in 
the theorem for Tij, 
= p-1 
Tij Eton Pm Jmn nj 
Nonlinear Term Projection 
The nonlinear terms are transformed as well, but the expres- 
sion cannot be easily written in matrix form Using the com- 
ponent form of the transformation, 
x i = Ej Pij vj , x - Ej Pij vj , vj - E k p-ljk x k , and 
substituting into the Cartesian normal form, gives: 
x i = (-'+1) Ej Pij (Ek p-ljk Xk) 4- Ej Pij Ek Jjk (El P-lkl xi) 
p-1 Aj ( E m - Xm) ( E p-1 
+ Ej Pij (Ek jk xk) Ei i P 11m n in Xn) 
Rearranging the orders of summation gives, 
x i = (--+1) E k (Ej Pij p-ljk) x k 4- E 1 (E k Ej Pij Jjk P-lkl) Xl 
p-1 
+ E n E m E k (E 1 Ej Pij jk Aji P-im P-lin) Xk Xm Xn 
Finally, 
indices 
performing the bracketed summations and relabeling 
gives us the network of the theorem, 
x i = -  x i + Ej Tij xj + Ejk 1 Tijkl Xj x k X 1 
with the expression for the tensor of the nonlinear term, 
466 Baird 
= p-1 . p-1 
Tijkl Emn Pim Amn mj P-lnk nl 
Q.E.D. 
LEARNING RULE EXTENSIONS 
This is the core of the mathematical story, and it may be ex- 
tended in many ways. When the columns of P ore orthonormal, 
then p-1 = pT and the formula above for the linear network 
coupling becomes T = pjpT. Then, for complex eigenvectors, 
Tj = E sx s xj s [cos(e s - ej s) + w s sin (e s - ejS)]. 
This is now a local, additive, incremental learning rule for 
synapse ij, and the system con be truly self-organizing be- 
cause the net con modify itself based on its own activity. 
Between units of equal phase, or when sis = ej s = 0 for o static 
pattern, this reduces to the usual Hebb rule. 
In a similar fashion, the learning rule for the higher order 
nonlinear terms becomes o multiple periodic outer product rule 
when the matrix A is chosen to hove o simple form. Given our 
present ignorance of the full biophysics of introcellulor 
processing, it is not entirely impossible that some dimension- 
olity of the higher order weights in the mathematical network 
could be implemented locally within the cells of o biological 
network, using the information available on the primary lines 
given by the linear connections discussed above. When the A 
matrix is chosen to hove uniform entries Aij = C for oll its 
off-diagonal 2 x 2 blocks, and uniform entries Ai = c - d 
for the diagonal blocks, then, 
= Ci,- 1 -dEs n/2 x- s x. s x? x- s rcose. s cose, s cosek s cosels 
Tijkl j K i j K I i j 
+ sine. s sine. s cose_ s cose. s + cose. s cose. s sinekS sinei s 
i _ j _ K _ i _ i j 
+ sinei s sine? sinek s sineiS]. 
This reduces to the multiple outer product 
Tikl C i kl d Es n/2 for static patterns. 
= - XlS Xj s Xk s Xl s , 
The network architecture generated by this learning rule is 
x i - - x 1 + Ej Tij xj + c x t Ej xj 2 - d x i Ejk i Tijkl xj x k x 1 
This reduces to an architecture without higher order correla- 
tions in the case that we choose o completely uniform A matrix 
(Al3 = c , for oll i,j). Then 
X 1 = -  X i 
Ej Tij xj + c x i Ej xj z 
A Bifurcation Theory Approach to Programming 467 
This network has fixed points on the axes of the normal form 
os always, but the stability condition is not satisfied since 
the diagonal normal form coefficients ore equal, not less, 
than the remaining A matrix entries. In (Baird 89) we describe 
how clamped input (inspiration) con break this symmetry and 
make the nearest stored pattern be the only attractor. 
All of the above results hold os well for networks with sig- 
maids, provided their coupling is such that they hove o Toyl- 
ot's expansion which is equal to the above networks up to 
third order. The results then hold only in the neighborhood of 
the origin for which the truncated expansion is accurate. The 
expected performance of such systems has been verified in 
simulations. 
Acknowledgements 
Supported by AFOSR-87-0517. I am very grateful for the support 
of Walter Freeman and invaluable assistance of Morris Hirsch. 
References 
B. Baird. Bifurcation Theory Methods For Programming Static 
or Periodic Attractors and Their Biurcations in Dynamic 
Neural Networks. Proc. IEEE Int. Conf. Neural Networks, 
San Diego, Ca.,pI-9, July(1988). 
B. Baird. Bifurcation Theory Approach to Vectorfield Program- 
ming for Periodic Attractors. Proc. INNS[IEEE Int. Conf. 
on Neural Networks. Washington D.C., June(lgBg). 
W. J. Freeman & B. Baird. Relation of 01factory EEG to Be- 
havior: Spatial Analysis. Behavioral Neuroscience (1988). 
W. . Freeman & B. W. van Dijk. Spatial Patterns of Visual 
Cortical EEG During Conditioned Reflex in o Rhesus Monkey. 
8rain Research, 422, p287(1987). 
C. M. Grey and W. Singer. Stimulus Specific Neuronal 
Oscillations in Orientation Columns of Cot Visual Cortex. 
PNAS. In Press(1988). 
Z. Li & J.J. Hopfield. Modeling The Olfactory Bulb. Biologi- 
cal Cybernetics. Submitted(1989). 
