Information Theoretic Analysis of 
Connection Structure from Spike Trains 
Satoru Shiono* 
Central Research Laboratory 
Mitsubishi Electric Corporation 
Amagasaki, Hyogo 661, Japau 
Satoshi Yamada 
Central Research Laboratory 
Mitsubishi Electric Corporation 
Amagasaki, Hyogo 661, Japan 
Michio Nakashima 
Central Research Laboratory 
Mitsubishi Electric Corporation 
Amagasaki, Hyogo 661, Japau 
Kenji Matsumoto 
Fculty of Pharmaceutical Science 
Hokkidou University 
Sapporo, Hokkaidou 060, Japau 
Abstract 
We have attempted to use information theoretic quantities for aa- 
lyzing neuronal connection structure from spike trains. Two point 
mutual information and its maximum value, channel capacity, be- 
tween a pir of neurons were found to be useful for sensitive de- 
tection of crosscorrelation and for estimation of synaptic strength, 
respectively. Three point mutual information among three neurons 
could give their interconnection structure. Therefore, our informa- 
tion theoretic analysis was shown to be a very powerful technique 
for deducing neuronal connection structure. Some concrete exam- 
ples of its application to simulated spike trMns are presented. 
1 INTRODUCTION 
The deduction of neuronal connection structure from spike trains, including synaptic 
strength estimation, has long been one of the central issues for understanding the 
structure and function of the neuronal circuit and thus the information processing 
*corresponding author 
515 
516 Shiono, Yamada, Nakashima, and Matsumoto 
mechanism at the neuronal circuitry level. A variety of crosscorrelational techniques 
for two or more neurons have been proposed axtd utilized (e.g., Melssen aztd Epping, 
1987; Aertsen et. al., 1989). There are, however, some difficulties with those 
techniques, as discussed by, e.g., Yang and Shamma (1990). It is sometimes difficult 
for the method to distinguish a significant crosscorrelation from noise, especially 
when the amount of experimental data is limited. The quantitative estimation 
of synaptic connectivity is another difficulty. And it is impossible to determine 
whether two neurons are directly connected or not, only by finding a significant 
crosscorrelation between them. 
The information theory has been shown to afford a powerful tool for the description 
of neuronal input-output relations, such as in the investigation on the neuronal cod- 
ing of the visual cortex (Eckhorn et. al., 1976; Optican and Richmond, 1987). But 
there has been no extensive study to apply it to the correlational analysis of action 
potential trains. Because a correlational method using information theoretic quan- 
tities is considered to give a better correlational measure, the information theory is 
expected to offer a unique correlational method to overcome the bove difficulties. 
In this paper, we describe information theory-based correlational aztalysis for action 
potential trains, using two and three point mutual information (MI) and chamnel 
cpacity. Because the information theoretic analysis by two point MI and channel 
cpacity will be published in near future (Yamada et. al., 1993a), more detMled de- 
scription is given here on the amalysis by three point MI for infering the relationship 
mong three neurons. 
2 
CORRELATIONAL ANALYSIS BASED ON 
INFORMATION THEORY 
2.1 INFORMATION THEORETIC QUANTITIES 
According to the information theory, the n point mutual information expresses the 
amount of information shared among n processes (McGill, 1955). Let X, Y and 
Z be processes, and t and s be the time delays of X and Y from Z, respectively. 
Using Shannon entropies H, two point MI between X and Y and three point MI, 
are defined (Shannon, 1948; Ikeda et. al., 1989): 
_(X: ) = H(X) + H(L) - H(X,, ), (1) 
I(X, : Y : Z) = H(X,) + H(Y)+ H(Z) - H(X,,Y,) 
-H(Y,Z) - H(Z,X,) + H(X,,Y,,Z). (2) 
I(X, : Y, : Z) is related to I(X, : Y,) as follows: 
r(x, : : z)= r(x, : v,)- r(x, : V, lZ), (3) 
where I(X: YslZ) means the two point conditional MI between X and Y if the 
state of Z is given. On the other hand, channel capacity is given by (r = s - t), 
CC(X: Yr)= maxI(X: Yr). (4) 
p(x,) 
We consider now X, Y and Z to be neurons whose spike activity has been measured. 
Information Theoretic Analysis of Connection Structure from Spike Trains 517 
Two point MI and two point conditional MI are obtained by (i, j, k = 0, 1), 
p(yj,,-[xi) 
I(X : Y,-) -- Ep(yj,,-]x,)p(x,)log p(yj,,.) , (5) 
i,j 
P(X"t' YJ'slzl ) (6) 
I(Xt : YslZ) - E p(xi,, yj,lzl )p(zl ) log p(x,,lzl )p(yj,lzl  ). 
i ,j,k 
where x, y and z mean the states of neurons, e.g., xx for the firing state and 
x0 for the non-firing state of X, and p( ) denotes probability. And three point 
MI is obtained by using Equation (3). Those information theoretic quantities are 
calculated by using the probabilities estimated from the spike trains of X, Y and Z 
after the spike trains are converted into time sequences consisting of 0 and 1 with 
discrete time steps, as described elswhere (Yamada et. al., 1993a). 
2.2 
PROCEDURE FOR THREE POINT MUTUAL INFORMATION 
ANALYSIS 
Suppose that a three point MI peak is found at (to, so) in the t, s-plane (see Figure 1). 
The three time delays, t0, so d r = so -to, are obtained. They are supposed to be 
time delays in three possible interconnections between any pair of neurons. Because 
the peak is not significant if only one pMr of the three neurons is interconnected, two 
or three of the possible interconnections with corresponding time delays should trnly 
work to produce the peak. We will utilize I(n : m) and I(n : roll ) (n, m, I = X, Y or 
Z) at the peak to find working interconnections out of them. These quantities are 
obtained by recalcnlating each probability in Equations (5) and (6) over the whole 
peak region. 
If two neurons, e.g., X and Y, are not interconnected either I(X: Y) or I(X: Y[Z) 
is equal to zero. The reverse proposition, however, is not true. The necessary 
and snfllcient condition for having no interconnection is obtained by calculating 
I(n: m) and I(n: roll ) for all possible interconnection structures. The neurons are 
rearranged and renamed A, B and C in the order of the time delays. There are only 
four interconnection structures, as shown in Table 1. 
I: No interconnection between A and B. A and B are statistically independent, i.e., 
p(a,,bj): p(ai)p(bj), I(A: B)= 0. The three point MI peak is negative. 
II: No interconnection between A and C. The states of A and C are statistically in- 
dependent when the state of B is given, i.e., p(ai, cklbj) = p(ailbj)p(cklbj), 
I(A: C[B) = 0. The peak is positive. 
III: No interconnection between B and C. Similar to case II, because p(bj, c ]ai) = 
p(bj[ai)p(ca[ai), I(B: cla): 0. The peak is positive. 
IV: Three interconnections. The above three cases are considered to occur concoIni- 
tautly in this case. The pek is positive or negative, depending on their 
relative contributions. Because A and B should have an apparent effect on 
the firing-probability of the postsynaptic neurons, I(A: B), I(A: C[B) 
and I(B: C[A) are all non-zero except for the case where the ativity of 
B completely coincides with that of A with the specified time delay (in 
this case, both I(A: CIB) and I(B: CIA ) are zero (see Vamada et. al., 
1993b)). 
518 Shiono, Yamada, Nakashima, and Matsumoto 
Table 1. Interconnection Structure and Information Theoretic Quantities 
Interconnection 
Structure 
2 point MI 
I(A:B) = 0 > 0 > 0 > 0 
I(A:C) _0 >0 >0 _0 
I(B:C) _0 >0 >0 _0 
2 point condition MI 
I(A:B [C) > 0 _ 0 -- 0 -- 0 
I(A:C [ B) > 0 = 0 -- 0 > 0 
I(B:C I A) > 0 _ 0 = 0 > 0 
3 point MI 
I(A:B:C) _ _ - - or 
From what we have described above, the interconnection structure for a three point 
MI peak is deduced utilizing the following procedure; 
(a) A negative 3pMI peak: it corresponds to case I or IV. The problem is to 
determine whether A and B are interconnected or not. 
(1) If I(A: B) = 0, case I. 
(2) If I(A: B) > 0, case IV. 
(b) A positive 3pMI peak: it corresponds to case II, III or IV. The existence of the 
A-G and B-G interconnections has to be checked. 
(1) 
(2) 
(4) 
I(A : CIB) > 0 and I(B : CIA ) > 0, case IV. 
I(A : CIB) = o and I(B : CIA) > 0, cse II. 
If I(A: C[B) > 0 and I(B: C[A) = 0, case III. 
If I(A: C[B) = 0 and I(B: C[A) = 0, the interconnection structure 
cannot be deduced except for the A-B interconnection. 
This procedure is applicable, if all the time delays are non-zero. If otherwise, some 
of the interconnections cannot be determined (Yamada et. al., 1993b). 
3 SIMULATED SPIKE TRAINS 
In order to characterize our information theoretic analysis, simulations of neu- 
ronal network models were carried out. We used a model neuron described by 
Information Theoretic Analysis of Connection Structure from Spike Trains 519 
the Hodgkin-Huxley equations (Yamadd et. al., 1989). The used equations and pa- 
rameters were described (YamdAd et. aL, 1993a). The Hodgkin-Huxley equations 
were mathematically integrated by the Runge-Kutta-Gill technique. 
4 RESULTS AND DISCUSSION 
4.1 
ANALYSIS BY TWO POINT MUTUAL INFORMATION AND 
CHANNEL CAPACITY 
The performance was previously reported of the information theoretic analysis by 
two point MI and channel capacity (Yamadd et. al., 1993a). 
Briefly, this anlytical method was compared with some conventional ones for both 
excitatory and inhibitory connections using action potential trains obtained by the 
simulation of a model neuronal network. It was shown to have the following ad- 
vantages. First, it reduced correlational measures within the bounds of noise nd 
simultaneously amplified beyond the bounds by its nonlinear function. It should be 
easier in its crosscorrelation graph to find a neuron pair having a weak but signif- 
icant interaction, especially when the synaptic strength is small or the mount of 
experimental data is limited. Second, channel capacity was shown to allow fairly 
effective estimation of synaptic strength, being independent of the firing probability 
of a presynaptic neuron, as long as this firing probability was not large enough to 
have the overlap of two successive postsynaptic potentials. 
4.2 ANALYSIS BY THREE POINT MUTUAL INFORMATION 
The practical application of the analysis by three point MI is shown below in detail, 
using spike trains obtained by simulation of the three-neuron network models shown 
in Figures i and 2 (Yamadd et. al., 1993b). 
The network model in Figure 1(1) has three interconnections. In Figure 1(2), three 
point MI has two positive peaks at (17ms, 12ms) (unit "ms" is omitted hereafter) 
and (17,30), and one negative peak at (0,12). For the peak at (17,12), the neurons 
are renamed A, B and C from the time delays (Z as A, Y as B and X as C), as 
in Table 1. Because only I(B: CIA ) '- 0 (see Figure 1 legend), the peak indicates 
case III with A---}B (Z---}Y) (s = 12) and AC (Z---}X) (t = 17)interconnections. 
Similarly, the peak at (17, 30) indicates Z---}X and X---}Y (s -t = 13) intercon- 
nections, and the peak at (0, 12) indicates Z--Y and X---}Y interconnections. The 
interconnection structure deduced from eax:h three point MI peak is consistent with 
eax:h other, and in agreement with the network model. 
Alternatively, the three point MI graphical presentation such as shown in Figure 
1(2) itself gives indication of some truly existing interconnections. If more than two 
three point MI peaks are found on one of the three lines, t = to, s = so and s-t = to, 
the interconnection with the time delay represented by this line is considered to be 
real. For exarnple, because the peaks at (17, 12) and (17, 30) are on the line of t = 17 
(Figure 1(2)), the interconnection represented by t = 17 (Z--,,X) are considered to 
be real. In a similar manner, the interconnections of s = 12 (Z--,,Y) and s - t = 12 
(X--,,Y) axe obtained. But this graphical indication is not complete, and thus the 
calculation of two point MI's and two point conditional MI's should be always 
520 Shiono, Yamada, Nakashima, and Matsumoto 
(1) 
Neurn X % Neuron  
Neuron Z 
(2) 
s (ms) 
t=17 
s-t=12 
2 
3pMI 
(bit) 
-0.0010 
0 
t (ms) so 
Figure 1. Three point MI analysis of simulated spike trains. (1) A three-neuron 
network model with Z--*X Z--*Y and X--,Y interconnections. The total number of 
spikes; X:4000, Y:5400, Z:3150. (2) Three point MI analysis of spike trains. Three 
point MI has two positive peaks at (17, 12) axtd (17, 30), and one negative peak at 
(0,12). For the peak at (17,12) the neurons are renamed (Z as A, Y as B and X 
as G). Two point MI and two point conditional MI for the peak at (17, 12) are: 
I(A: B) = 0.03596, I(A: C) = 0.06855, I(B: C) = 0.01375, I(A: BIc)- o.02126, 
I(A : C[B) = 0.05376, I(B : CIA) = 0.00011. So, I(B : CIA) -' O, indlcting case 
III (see Table 1) with A--*B (Z--*Y) and A--,C (J-,X)interconnections. Similarly, 
for the peaks at (17, 30)and at (0, 12), Z--,X and X--*Y interconnections, and Z--*Y 
nd X-Y interconnections are obtained, respectively. 
performed for contirmation. 
The network model in Figure 2(1) has four interconnections. Three point MI h 
ave major peaks: four positive peaks at (17,-12), (17,30), (-24,-12) and (17,12) 
nd one negative peak at (0, 10). The peaks at (17,-12), (17,12) and (17,30) are 
on the line of t = 17 (Z--,X), the peaks at (17,-12) and (-24,-12) are on the 
line of s = -12 (Z-Y), the peaks at (17,12) and (0, 10) are on the line of s = 12 
(Z--,Y), and the peaks at (-24,-12), (0, 10) and (17,30) areon the line of 
Information Theoretic Analysis of Connection Structure from Spike Trains 521 
(1) 
Neuron X C Neuron y 
Neuron Z 
(2) 
s (ms) 
t=17 
s-t=12 
=12 
3pMI 2 
(bit) 
o 
t (ms) 
$o 
Figure 2. Three point MI analysis of simulated spike trains. (1) A three-neuron 
network model with Z---}X Z---}Y, Z,-Y and XY interconnections. The total 
number of spikes; X:4300, Y:5150, Z:4850. (2) Three point MI amalysis of spike 
trains. Three point MI has five major peaks, four positive peaks at (17,-12), 
(17,12), (17,30) and (-24,-12), and one negative peak at (0, 10). 
s - t = 12 (XY). The calculation of two point MI and two point conditional MI 
for each peak gives the confirmation that each three point MI peak was produced 
by two interconnections. Namely, their calculation indicates Z--,X 
(s = -12), Z---}Y (s: 12) and XY (s- t: 12) interconnections. There are 
also some small peaks. They axe considered to be ghost peaks due to two or three 
interconnections, at least one of which is a combination of two interconnections 
found by analyzing the major peaks. For example, the positive peak at (-7,-12) 
indica. tes Z,--Y and X--*Y interconnections, but the latter (s -  = -5) is the 
combination of the Z--*X interconnection (t = 17) and the Z--*Y interconnection 
(s = 12). 
The interconnection structure of a network containing n inhibitory interconnection 
or consisting of more than four neurons can also be deduced, although it becomes 
more difficult to perform the three point MI analysis. 
522 Shiono, Yamada, Nakashima, and Matsumoto 
References 
A.M. H. J. Aertsen, G. L. Gerstein, M. K. Habib & G. Palm. (1989) Dynamics of 
nenronal firing correlation: modulation of "effective connectivity". J. NeurolohyioL 
61: 900-917. 
R. Eckhorn, O. J. Griisser, J. Kri511er, K. Pellnitz & B. Piipel. (1976) Efficiency 
of different nenronal codes: information transfer calculations for three different 
nenronal systems. Biol. Gybern. 22: 49-60. 
K. Ikeda, K. Otsnka & K. Matsumoto. (1989) Maxwell-Bloch turbulence. Prog. 
Theor. Phys., Suppl. 99: 295-324. 
W. J. McGill. (1955) Multivariate information transmission. lite Trans. In.L 
Theory 1: 93-111. 
W. J. Meissen & W. J. M. Epping. (1987) Detection and estimation of neural 
connectivity based on crosscorrelation analysis. Biol. Gybern. 5?: 403-414. 
L. M. Optican & B. J. Richmond. (1987) Temporal encoding of two-dimensional 
patterns by single nnits primate inferior temporal cortex. III. Information theoretic 
analysis. J. Neurolohysiol. 5?: 162-178. 
C. E. Shannon. (1948) A mathematical theory of communication. Bell. Syst. 
Techn. J. 2?: 379-423. 
S. Yamada, M. Nakashima, K. Matsumoto & S. Shiono. (1993a) Information the- 
oretic analysis of action potential trMns: I. Analysis of correlation between two 
neurons. Biol. Cybern., in press. 
S. Yamada, M. Nakashima, K. Matsumoto & S. Shiono. (1993b) Information the- 
oretic analysis of action potential trains: II. Analysis of correlation among three 
neurons. submitted to Biol. Cybern. 
W. M. YamaAa, C. Koch & P. R. Adams. (1989) Multiple channels and calcium 
dynamics. In C. Koch & I. Segev (ed), Methods in Neuronal Modeling: From 
Synapses to Neurons, 97-133, Caanbridge, MA, USA: MIT Press. 
X. Yang & S. A. Sharnma. (1990) Identification of connectivity in neural networks. 
Biophys. J. 5?: 987-999. 
