Discontinuous Recall Transitions Induced By 
Competition Between Short- and Long-Range 
Interactions in Recurrent Networks 
N.S. Skantzos, C.F. Beckmann and A.C.C. Coolen 
Dept of Mathematics, King's College London, Strand, London WC2R 2LS, UK 
E-mail: skantzos@mth.kcl.ac.uk tcoolen@mth.kcl.ac.uk 
Abstract 
We present exact analytical equilibrium solutions for a class of recur- 
rent neural network models, with both sequential and parallel neuronal 
dynamics, in which there is a tunable competition between nearest- 
neighbour and long-range synaptic interactions. This competition is 
found to induce novel coexistence phenomena as well as discontinuous 
transitions between pattern recall states, 2-cycles and non-recall states. 
1 INTRODUCTION 
Analytically solvable models of large recurrent neural networks are bound to be simplified 
representations of biological reality. In early analytical studies such as [ 1, 2] neurons were, 
for instance, only allowed to interact with a strength which was independent of their spatial 
distance (these are the so-called mean field models). At present both the statics of infinitely 
large mean-field models of recurrent networks, as well as their dynamics away from satura- 
tion are well understood, and have obtained the status of textbook or review paper material 
[3, 4]. The focus in theoretical research of recurrent networks has consequently turned to 
new areas such as solving the dynamics of large networks close to saturation [5], the anal- 
ysis of finite size phenomenology [6], solving biologically more realistic (e.g. spike-based) 
models [7] or analysing systems with spatial structure. In this laper we analyse mod- 
els of recurrent networks with spatial structure, in which there are two types of synapses: 
long-range ones (operating between any pair of neurons), and short-range ones (operating 
between nearest neighbours only): In contrast to early papers on spatially structured net- 
works [8], one here finds that, due to the nearest neighbour interactions, exact solutions 
based on simple mean-field approaches are ruled out. Instead, the present models can be 
solved exactly by a combination of mean-field techniques and the so-called transfer ma- 
trix method (see [9]). In parameter regimes where the two synapse types compete (where 
one has long-range excitation with short-range inhibition, or long-range Hebbian synapses 
with short-range anti-Hebbian synapses) we find interesting and potentially useful novel 
phenomena, such as coexistence of states and discontinuous transitions between them. 
338 N. S. Skantzos, C. F. Beckmann and A. C. C. Coolen 
2 MODEL DEFINITIONS 
We study models with N binary neuron variables ai = 4-1, which evolve in time stochas- 
tically on the basis of post-synaptic potentials hi (), following 
I [1 :k: tanh[/hi((t))]] hi(W) = E Jijaj + Oi (1) 
Prob[ai(t + 1)= +1] =  
The variables Jq and Oi represent synaptic interactions and firing thresholds, respectively. 
The (non-negative) parameter/ controls the amount of noise, with/ = 0 and/ = oo corre- 
sponding to purely random and purely deterministic response, respectively. If the synaptic 
matrix is symmetric, both a random sequential execution and a fully parallel execution of 
the stochastic dynamics (1) will evolve to a unique equilibrium state. The corresponding 
microscopic state probabilities can then both formally be written in the Boltzmann form 
Po() ~ exp[-BH()], with [10] 
Zseq() ---- -- E 'iZij'J-E OiO'i 
i<j  
1 
Hpar() = - E log cosh[/hi()]- E Oiai 
i i 
(2) 
For large systems the macroscopic observables of interest can be obtained by differentiation 
of the free energy per neuron f = - limv- (/N) - log Ye exp[-/H()], which acts 
as a generating function. For the synaptic interactions JO and the thresholds Oi we now 
make the following choice: 
model I: Ji =  ij + Js(Sid+! + 5i,j-! ) ij Oi = Oi (3) 
(which corresponds to the result of having stored a binary pattern '  {-1, 1} v through 
Hebbian-type learning), with Jr, J,, 0   and i + N -- i. The neurons can be thought of as 
being arranged on a periodic one-dimensional array, with uniform interactions of strength 
Jti IN, in combination with nearest neighbour interactions of strength J,ij. Note that 
model I behaves in exactly the same way as the following 
model II: jq _ Jt 
-  + J(5,j+ + 5i,j_) 0 = 0 (4) 
since a simple transformation ai ---> aii maps one model into the other. Taking derivatives 
of f with respect to the parameters 0 and J, for model II produces our order parameters, 
expressed as equilibrium expectation values. For sequential dynamics we have 
Of _ lim 1 E(rri+rri ) (5) 
Of_ lim 1 
m = O0 N-o 
For parallel dynamics the corresponding expressions turn out to be 
1Of_ lim I 1 Of _ lim 1 E(ai+x tanh[/3hi(ff)]) 
m-- 200 N-oN 
(6) 
We have simplified (6) with the identities (ai+ tanh[/3hi()]} = (ai- tanh[/hi()]} and 
(tanh[/hi ()]} = (ai}, which follow from (1) and from invariance under the transforma- 
tion i --> N + 1 - i (for all i). For sequential dynamics a describes the average equilibrium 
state covariances of neighbouring neurons. For parallel dynamics a gives the average equi- 
librium state covariances of neurons at a given time t, and their neighhours at time t + 1 (the 
difference between the two meanings of a will be important in the presence of 2-cycles). 
In model II m is the average activity in equilibrium, whereas for model I one finds 
m = lim 1 
i 
This is the familiar overlap order parameter of associative memory models [1, 2], which 
measures the quality of pattern recall in equilibrium. The observable a transforms similarly. 
Competition between Short- and Long-Range Interactions 339 
3 SOLUTION VIA TRANSFER MATRICES 
From this stage onwards our analysis will refer to model II, i.e eqn (4); the results can im- 
mediately be translated into the language of model I (3) via the transformation cr i -- crii. 
In calculating f it is advantageous to separate terms induced by the long-range synapses 
from those induced by the short-range ones, via insertion of 1 = fdm 6[rn -  Y]i cri]' 
Upon using the integral representation of the 6-function, we then arrive at 
f = - lim 1 
with v- - log dmdrh e -v(''r) 
I 2 1 
qseq(m,r) = -im -mO - Jtm /3N lOgRseq() (7) 
1 
qSpar (m, rh) =-im& - m0 -/30 log Rpar(m, rh) (8) 
The quantities R contain all complexities due to the short-range interactions in the model 
(they describe a periodic one-dimensional system with neighbout interactions only): 
aseq() = W' E, 
{-,} 
Rpar(m, ) - E -i/3Vn 'i rl 'ilogcosh/3[Jtrn+O+Jo(ri+x+a,_x)] 
e{-,}  
They can be calculated using the transfer-matrix technique [9], which exploits an interpre- 
tation of the summations over the N neuron states ai as matrix multiplications, giving 
: [Tseq] Tseq: e_/jo e/Jo+i/ 
= [ror] ror = coh[o] 
where wo =Jtm +  and w+ = wo 4- 2J. The identity Tr [T v] = A v A v 
+ + _,in which 
A+ are the eigenvalues of the 2 x 2 matrix T, enables us to take the limit N   in our 
equations. The integral over (m, h) is for N   evaluated by gradient descent, and is 
dominated by the saddle points of the exponent b. We thus arrive at the transparent result 
lseq 
f = extr (m,h) 4r(m, h) = -imh- mO-  logA r (9) 
where seq and lpar 
.,+ .,+ are the largest eigenvalues of Tseq and Tpar. For simplicity, we will 
restrict ourselves to the case where 0 = 0; generalisation of what follows to the case of 
arbitrary 0, by using the full form of (9), is not significantly more difficult. The expressions 
defining the value(s) of the order parameter rn can now be obtained from the saddle point 
equations Oraq(m, ) -- Oq(m, ) -- 0. Straightforward differentiation shows that 
sequential: rh = imJt, rn = G(rn; Jr, Js) 
parallel: rh = irnJt, rn = G (rn; Jr, J ) for Jt > 0 (10) 
 = -imJt, rn = G(rn;-Jt,-Js) for Jt < 0 
with 
G(ra; Jt, J ) = sinh[/Jtm] (11) 
sinh:[/Jtrn] + e-4/3Jo 
Note that equations (10,11) allow us to derive the physical properties of the parallel dy- 
namics model from those of the sequential dynamics model via simple transformations. 
340 N. S. Skantzos, C. F. Beckmann and.4. C. C. Coolen 
4 PHASE TRANSITIONS 
Our main order parameter m is to be determined by solving an equation of the form m - 
G(m), in which G(m) = G(m; Jr, Js) for both sequential and parallel dynamics with 
Jt >_ 0, whereas G(m) = G(m;-Jt,-Js) for parallel dynamics with Jt < 0. Note that, 
due to (7(0; Jr, J,) - 0, the trivial solution m = 0 always exists. In order to obtain a phase 
diagram we have to perform a bifurcation analysis of the equations (10,11), and determine 
the combinations of parameter values for which specific non-zero solutions are created or 
annihilated (the transition lines). Bifurcations of non-zero solutions occur when 
m = (7(m) and 1 = (7'(m) (12) 
The first equation in (12) states that m must be a solution of the saddle-point problem, the 
second one states that this solution is in the process of being created/annihilated. Nonzero 
solutions of m = G(m) can come into existence in two qualitatively different ways: as con- 
tinuous bifurcations away from the trivial solution m - 0, and as discontinuous bifurcations 
away from the trivial solution. These two types will have to be treated differently. 
4.1 Continuous Transitions 
An analytical expression for the lines in the (/J,,/Jt) plane where continuous transitions 
occur between recall states (where m  0) and non-recall states (where rn = 0) is obtained 
by solving the coupled equations (12) for m = 0. This gives: 
cont. trans.: sequential: Jt = e -23"" 
parallel: /3Jr = e -2a' and /3Jr = -e a' (13) 
If along the transition lines (13) we inspect the behaviour of (7(m) close to m = 0 we 
can anticipate the possible existence of discontinuous ones, using the properties of G(m) 
for m -> +o0, in combination with G(-m) = -G(m). Precisely at the lines (13) we 
1FJ-'"(0).m3 q- (.9(m5). Since lim,-oo G(m) = 1 one knows that 
have (7(m) = m + ,_. 
for (7'"(0) > 0 the function (7(m) will have to cross the diagonal (7(m) = m again at 
some value m > 0 in order to reach the limit (7(o0) - 1. This implies, in combination 
with (7(-m) = --G(m), that a discontinous transition must have already taken place earlier, 
and that away from the lines (13) there will consequently be regions where one finds five 
solutions of m = (7(m) (two positive ones, two negative ones). Along the lines (13) the 
condition (7'" (0) > 0, pointing at discontinuous transitions elsewhere, translates into 
 log 3 
sequential: /Jt > vf and /J < - 
(14) 
i log 3 
parallel: ]Jt[ >  and I/J,l < - 
4.2 Discontinuous Transitions 
In the present models it turns out that one can also find an analytical expression for the 
discontinuous transition lines in the (/Js,/Jt) plane, in the form of a parametrisation. For 
sequential dynamics one finds a single line, parametrised by z =/Jtm  [0, o0): 
 2 3 1 'tanh(x) sinh2 (x) ] 
discont. trans.: B Jr(x) = x-tanh(x) ' BJs(x) -- - log x-tanh(x) 
(15) 
 log 3 and/Jt (0) = vf, the discontin- 
Since this parametrisation (15) obeys/Js (0) = - 
uous transition indeed starts precisely at the point predicted by the convexity of (7(m) at 
m - 0, see (14). For sequential dynamics the line (15) gives all non-zero solutions of the 
coupled equations (12). For parallel dynamics one finds, in addition to (15), a second 'mir- 
ror image' transition line, generated by the transformation {/Jt,/J }  {-/Jt, -/J }. 
Competition between Short- and Long-Range Interactions 341 
5 PHASE DIAGRAMS 
o 
Iml>0, a>O 
-2 -1 0 1 2 2 
/3J /J 
I '..coex\ 
I '"X Iml>0, a>O 
'*,,. redpoi,, 
 2  m--O, a<O 
 /3Jr  
m, a>O  -2  1, a<O 
 2 -cycle 
-2 -1 0 1 
Figure 1: Left: phase diagram for sequential dynamics, involving three regions: (i) a region with 
m = 0 only (here a = l:anh[/SJs]), (ii) a region with two m - 0 fixed-point states (with opposite 
sign, and with identical a > 0), and (iii) a region where the m = 0 state and the two m -f: 0 
states coexist. The (i) - (ii) and (ii) --> (iii) transitions are continuous (solid lines), whereas the 
(i) --> (iii) transition is discontinuous (dashed line). Right: phase diagram for parallel dynamics, 
involving the above regions and transitions, as well as a second set of transition lines (in the region 
Jt < 0) which are exact reflections in the origin of the first set. Here, however, the m -- 0 region has 
a = tanh[2/3Js], the two m  0 physical solutions describe 2-cycles rather than fixed-points, and 
the Jt < 0 coexistence region describes the coexistence of an m = 0 fixed-point and 2-cycles. 
Having determined the transition lines in parameter space, we can turn to the phase dia- 
grams. A detailed expose of the various procedures followed to determine the nature of the 
various phases, which are also dependent on the type of dynamics used, goes beyond the 
scope of this presentation; here we can only present the resulting picture.  Figure 1 shows 
the phase diagram for the two types of dynamics, in the (/Js,13Jt) plane (note: of the 
three parameters {/, Js, Jr} one is redundant). In contrast to models with nearest neigh- 
bour interactions only (Jr = 0, where no pattern recall ever will occur), and to models with 
mean-field interactions only (J, = 0, where pattern recall can occur), the combination of 
the two interaction types leads to qualitatively new modes of operation. This especially in 
the competition region, where J > 0 and J, < 0 (Hebbian long-range synapses, com- 
bined with anti-Hebbian short range ones). The novel features of the diagram can play a 
useful role: phase coexistence ensures that only sufficiently strong recall cues will evoke 
pattern recognition; the discontinuity of the transition subsequently ensures that in the lat- 
ter case the recall will be of a substantial quality. In the case of parallel dynamics, similar 
statements can be made in the opposite region of synaptic competition, but now involving 
2-cycles. Since figure 1 cannot show the zero noise region (/ = T - = oo), we have also 
drawn the interesting competition region of the sequential dynamics phase diagram in the 
(Jr, T) plane, for J, = -1 (see figure 3, left picture). At T = 0 one finds coexistence of 
recall states (m 5 0) and non-recall states (m = 0) for any Jt > 0, as soon as J, < 0. 
In the same figure (right picture) we show the magnitude of the discontinuity in the order 
parameter m at the discontinuous transition, as a function of/Jt. 
Due to the occurrence of imaginary saddle-points in (10) and our strategy to eliminate the vari- 
able  by using the equation 0,b(rn, ) = 0, it need not be true that the saddle-point with the low- 
est value of b(rn, ) is the minimum of b (complex conjugation can induce curvature sign changes, 
and in addition the minimum could occur at boundaries or as special limits). Inspection of the sta- 
tus of saddle-points and identification of the physical ones in those cases where there are multiple 
solutions is thus a somewhat technical issue, details of which will be published elsewhere [ 11 ]. 
342 N. S. Skantzos, C. F. Beckmann and A. C. C. Coolen 
T 
6 
5 
4 
3 
2 
1 
0 
m--O, a<O , . 
? [ m>O, a>O 
2 4 6 8 10 
m 
1.0 
0.8 
0.6 
0.4 
0.2 t 
0.0 I 
0 
2 4 6 8 
Figure 2: Left picture: alternative presentation of the competition region of the sequential dynamics 
phase diagram in figure 1. Here the system states and transitions are drawn in the (Jr, T) plane 
(T = /-), for J8 = -1. Right picture: the magnitude of the 'jump' of the overlap m along the 
discontinuous transition line, as a function of/3Jr. 
The fact that for parallel dynamics one finds 2-cycles in the lower left corner of the phase 
diagram (figure 1) can be inferred from the exact dynamical solution available along the 
line Js = 0 (see e.g. [4]), provided by the deterministic map m(t + 1) = tanh[/Jtm(t)]. 
Finally we show, by way of further illustration of the coexistence mechanism, the value of 
reduced exponent qbseq(m) given in (9), evaluated upon elimination of the auxiliary order 
parameter : b(m) = qbseq(m, ira&). The result, for the parameter choice (/, Jr) = (2, 3) 
and for three different short-range coupling stengths (corresponding to the three phase 
regimes: non-zero recall, coexistence and zero recall) is given in figure 3. In the same 
figure we also give the sequential dynamics bifurcation diagram displaying the value(s) of 
the overlap rn as a function of 6Jt and for/Js = -0.6 (a line crossing all three phase 
regimes in figure (1)). 
0'01 ' ' i l'Ot ' ' ' 
,(m) m 0.0 
-1.5 
4.5 
-1.0 -0.5 0.0 0.5 1.0 2.0 2.5 3.0 4.0 
m lJt 
Figure 3: Left: Graph of the reduced exponent 4(m) = qbseq(m, irnJt) for the parameter choice 
(/3, Jr) = (2, 3). The three lines (from upper to lower: J, = -1.2,-0.8,-0.2) correspond to 
regimes where (i) m' - 0 only (ii) coexistence of trivial and non-trivial recall states occurs and 
(iii) m -- 0 only. Right: Sequential dynamics bifurcation diagram displaying for/J8 -- -0.6 the 
possible recall solutions. For a critical lJt given by (15) m jumps discontinuously to non-zero 
values. For increasing values of lJt the unstable m - 0 solutions converge towards the trivial one 
until/3Jr = exp(1.2) where a continuous phase transition takes place and m = 0 becomes unstable. 
Competition between Short- and Long-Range Interactions 343 
6 DISCUSSION 
In this paper we have presented exact analytical equilibrium solutions, for sequential and 
parallel neuronal dynamics, for a class of recurrent neural network models which allow for 
a tunable competition between short-range synapses (operating between nearest neighbours 
only) and long-range ones (operating between any pair of neurons). The present models 
have been solved exactly by a combination of mean-field techniques and transfer matrix 
techniques. We found that there exist regions in parameter space where discontinuous 
transitions take place between states without pattern recall and either states of partial/full 
pattern recall or 2-cycles. These regions correspond to the ranges of the network parameters 
where the competition is most evident, for instance, where one has strongly excitatory long- 
range interactions and strongly inhibitory short-range ones. In addition this competition is 
found to generate a coexistence of pattern recall states or 2-cycles with the non-recall state, 
which (in turn) induces a dependence on initial conditions of whether or not recall will at 
all take place. 
This study is, however, only a first step. In a similar fashion one can now study more com- 
plicated systems, where (in addition to the long-range synapses) the short-range synapses 
reach beyond nearest neighhours, or where the system is effectively on a two-dimensional 
(rather than one-dimensional) array. Such models can still be solved using the techniques 
employed here. A different type of generalisation would be to allow for a competition 
between synapses which would not all be of a Hebbian form, e.g. by having long-range 
Hebbian synapses (modeling processing via pyramidal neurons) in combination with short- 
range inhibitory synapses without any effect of learning (modeling processing via simple 
inhibitory inter-neurons). In addition, one could increase the complexity of the model by 
storing more than just a single pattern. In the latter types of models the various pattern 
components can no longer be transformed away, and one has to turn to the methods of 
random field Ising models (see e.g. [12]). 
References 
[8] 
[91 
[10] 
[11] 
[12] 
[1] D.J. Amit, H. Gutfreund and H. Sompolinsky (1985), Phys. Rev. A32, 1007-1018 
[2] D.J. Amit, H. Gutfreund and H. Sompolinsky (1985), Phys. Rev. Lett. 55, 1530-1533 
[3] A.C.C. Coolen and D. Sherrington (1993), in J.G.Taylor (editor) Mathematical Ap- 
proaches to Neural Networks, Elsevier Science Publishers, 293-306 
[4] A.C.C. Coolen (1997), Statistical Mechanics of Neural Networks, King's College 
London Lecture Notes 
[5] A.C.C. Coolen, S.N. Laughton and D. Sherrington (1996), in D.S. Touretzky, M.C. 
Mozer and M.E. Hasselmo (eds) Advances in Neural Information Processing Systems 
8, MIT Press 
[6] A. Castellanos, A.C.C. Coolen and L. Viana (1998), J. Phys. A 31, 6615-6634 
[7] E. Domany, J.L. van Hemmen and K. Schulten (eds) (1994), Models of Neural Net- 
works II, Springer 
A.C.C. Coolen and L.G.V.M. Lenders (1992), J. Phys A 25, 2593-2606 
J.M. Yeomans (1992), Statistical Mechanics of Phase Transitions, Oxford U.P. 
P. Peretto (1984), Biol. Cybern. 50, 51-62 
N.S. Skantzos and A.C.C. Coolen (1998), in preparation 
U. Brandt and W. Gross (1978); Z. Physik B 31,237-245 
