Regular and Irregular Gallager-type 
Error-Correcting Codes 
Y. Kabashima and T. Murayama 
Dept. of Compt. Intl. & Syst. Sci. 
Tokyo Institute of Technology 
Yokohama 2268502, Japan 
D. Saad and R. Vicente 
Neural Computing Research Group 
Aston University 
Birmingham B4 7ET, UK 
Abstract 
The performance of regular and irregular Gallager-type error- 
correcting code is investigated via methods of statistical physics. 
The transmitted codeword comprises products of the original mes- 
sage bits selected by two randomly-constructed sparse matrices; 
the number of non-zero row/column elements in these matrices 
constitutes a family of codes. We show that Shannon's channel 
capacity may be saturated in equilibrium for many of the regular 
codes while slightly lower performance is obtained for others which 
may be of higher practical relevance. Decoding aspects are con- 
sidered by employing the TAP approach which is identical to the 
commonly used belief-propagation-based decoding. We show that 
irregular codes may saturate Shannon's capacity but with improved 
dynamical properties. 
I Introduction 
The ever increasing information transmission in the modern world is based on re- 
liably communicating messages through noisy transmission channels; these can be 
telephone lines, deep space, magnetic storing media etc. Error-correcting codes play 
a significant role in correcting errors incurred during transmission; this is carried out 
by encoding the message prior to transmission and decoding the corrupted received 
code-word for retrieving the original message. 
In his ground breaking papers, Shannon[I] analyzed the capacity of communication 
channels, setting an upper bound to the achievable noise-correction capability of 
codes, given their code (or symbol) rate, constituted by the ratio between the num- 
ber of bits in the original message and the transmitted code-word. Shannon's bound 
is non-constructive and does not provide a recipe for devising optimal codes. The 
quest for more efficient codes, in the hope of saturating the bound set by Shannon, 
has been going on ever since, providing many useful but sub-optimal codes. 
One family of codes, presented originally by Gallager[2], attracted significant inter- 
est recently as it has been shown to outperform most currently used techniques[3]. 
Gallager-type codes are characterized by several parameters, the choice of which 
defines a particular member of this family of codes. Current theoretical results[3] 
Regular and Irregular Gallager-type Error-Correcting Codes 2 73 
offer only bounds on the error probability of various architectures, proving the ex- 
istence of very good codes under some restrictions; decoding issues are examined 
via numerical simulations. 
In this paper we analyze the typical performance of Gallager-type codes for several 
parameter choices via methods of statistical mechanics. We then validate the an- 
alytical solution by comparing the results to those obtained by the TAP approach 
and via numerical methods. 
2 The general framework 
In a general scenario, a message represented by an N dimensional Boolean vector 
 is encoded to the M dimensional vector jo which is transmitted through a noisy 
channel with some flipping probability p per bit (other noise types may also be 
studied). The received message J is then decoded to retrieve the original message. 
In this paper we analyze a slightly different version of Gallager-type codes termed 
the MN code[3] that is based on choosing two randomly-selected sparse matrices A 
and B of dimensionality M x N and M x M respectively; these are characterized 
by K and L non-zero unit elements per row and C and L per column respectively. 
The finite numbers K, C and L define a particular code; both matrices are known 
to both sender and receiver. Encoding is carried out by constructing the modulo 
2 inverse of B and the matrix B-1A (mod 2); the vector jo_ B-1A  (mod 2,  
Boolean vector) constitutes the codeword. Decoding is carried out by taking the 
product of the matrix B and the received message J- jo +4 (mod 2), corrupted 
by the Boolean noise vector 4, resulting in A+B. The equation 
A + B = AS + B' (mod 2) (1) 
is solved via the iterative methods of Belief Propagation (BP)[3] to obtain the most 
probable Boolean vectors $ and -; BP methods in the context of error-correcting 
codes have recently been shown to be identical to a TAP[4] based solution of a 
similar physical system[5]. 
The similarity between error-correcting codes of this type and Ising spin systems 
was first pointed out by Sourlas[6], who formulated the mapping of a simpler code, 
somewhat similar to the one presented here, onto an Ising spin system Hamiltonian. 
We recently extended the work of Sourlas, that focused on extensively connected 
systems, to the finite connectivity case[5] as well as to the case of MN codes [7]. 
To facilitate the current investigation we first map the problem to that of an Ising 
model with finite connectivity. We employ the binary representation (:hl) of the 
dynamical variables $ and - and of the vectors J and jo rather than the Boolean 
(0, 1) one; the vector jo is generated by taking products of the relevant binary 
message bits j0 __ I'Iie i, where the indices p - (il,...iK) correspond to the 
non-zero elements of B-1A, producing a binary version of jo. As we use statistical 
mechanics techniques, we consider the message and codeword dimensionality (N 
and M respectively) to be infinite, keeping the ratio between them R - N/M, 
which constitutes the code rate, finite. Using the thermodynamic limit is quite 
natural as Gallager-type codes are usually used for transmitting long (10 4- 10 5) 
messages, where finite size corrections are likely to be negligible. To explore the 
system's capabilities we examine the Hamiltonian 
274 Y. Kabashima, T. Murayama, D. Saad and R. Ficente 
The tensor product 7, where 7 = I'Ie,  I'Ije,, j and a = (j,... j), is 
the binary equivalent of A+B, treating both signal (S and index i) d noise 
(w and index j) simultaneously. Elements of the sparse connectivity tensor 
take the value I if the corresponding indices of both signal and noise are chosen 
(i.e., if all corresponding indices of the matrices A and B are 1) d 0 otherwise; 
it h C unit elements per /-index d L per j-index representing the system's 
degree of connectivity. The 5 function provides I if the selected sites' product 
ie Si je rj is in disagreement with the corresponding element , recording 
an error, d 0 otherwise. Notice that this term is not frustrated,  there are 
M+N degrees of freedom and only M constrnts from Eq.(1), and can therefore 
vish at suciently low temperatures. The lt two terms on the right represent 
our prior knowledge in the ce of sparse or bied messages F d of the noise 
level F and require signing certain values to these additive fields. The choice of 
   imposes the restriction of Eq.(1), limiting the solutions to those for which 
the first term of Eq.(2) vanishes, while the lt two terms, scaled with , survive. 
Note that the noise dynic variables r are irrelevant to menuring the retrieval 
 ) The latter monitors the normized mean 
success m =  E i sign<Si)  
overlap between the Bayes-optim retrieved message, shown to correspond to the 
alignment of <Si) to the nearest binary value[6], d the original message; the 
subscript  denotes therm averaging. 
Since the first part of Eq.(2) is invariant under the map Si  Sii, wj  wiQ and 
   ie i je Q = 1, it is useful to decouple the correlation between the 
vectors S, w and , . Rewriting Eq.(2) one obtns a similar expression apart from 
the lt terms on the right which become Fs/  Sn  and Fr/   . 
The random selection of elements in  introduces disorder to the system which 
is treated via methods of statistical physics. More specifically, we calculate the 
partition function Z(, J) = {S,w} exp[-] averaged over the disorder and the 
statistical properties of the message and noise, using the replica method[5, 8, 9]. 
Taking    gives rise to a set of order parameters 
I 1 
i=1 tm i=1 lm 
(2) 
where a, B, .. represem replica indices, and the variables Zi d  come from 
enforcing the restriction of C and L connections per index respectively[5]: 
5 - C = , (3) 
and similarly for the restriction on the j indices. 
To proceed with the calculation one has to make  sumption about the order 
parameters symmetry. The sumption made here, and validated later on, is that 
of replica symmetry in the following representation of the order parameters and the 
related conjugate variables 
q,a.., = aq / aX x t = a?/ 
, 
(4) 
where 1 is the number of replica indices, a. are normalization coefficients, and 
(x), (:), p(y) and () represent probability distributions. Unspecified integrals 
Regular and Irregular Gallager-type Error-Correcting Codes 2 75 
are over the range [-1, +1]. One then obtains an expression for the free energy 
per spin expressed in terms of these probability distributions 1/N (ln Z,,v The 
free energy can then be calculated via the saddle point method. Solving the 
equations obtained by varying the free energy w.r.t the probability distributions 
'(x), (), p(y) and (9), is difficult as they generally comprise both delta peaks 
and regular[9] solutions for the ferromagnetic and paramagnetic phases (there is no 
spin-glass solution here as the system is not frustrated). The solutions obtained 
in the case of unbiased messages (the most interesting case as most messages are 
compressed prior to transmission) are for the ferromagnetic phase: 
'(x) = 5(x- 1) , () - 5( - 1) , p(y) - 5(y - 1) , (9) = 5(9 - 1) , (5) 
and for the paramagnetic phase: 
(x) = 5(x), ()=5(), (9)=5(9) 
1 + tanh F 1 - tanh F 
P(Y) - 2 5(y - tanh F) + 2 5(y + tanh F) . (6) 
These solutions obey the saddle point equations. However, it is unclear whether 
the contribution of other delta peaks or of an additional continuous solution will be 
significant and whether the solutions (5) and (6) are stable or not. In addition, it 
is also necessary to validate the replica symmetric ansatz itself. To address these 
questions we obtained solutions to the system described by the Hamiltonian (2) via 
TAP methods of finitely connected systems[5]; we solved the saddle point equations 
derived from the free energy numerically, representing all probability distributions 
by up to 10 4 bin models and by carrying out the integrations via Monte-Carlo 
methods; finally, to show the consistency between theory and practice we carried 
out large scale simulations for several cases, which will be presented elsewhere. 
3 Structure of the solutions 
The various methods indicate that the solutions may be divided to two different 
categories: K- L- 2 and either K _> 3 or L _> 3. We therefore treat them separately. 
For unbiased messages and either K _> 3 or L _> 3 we obtain the solutions (5) and 
(6) both by applying the TAP approach and by solving the saddle point equations 
numerically. The former was carried out at the value of F which corresponds to 
the true noise and input bias levels (for unbiased messages Fs = 0) and thus to 
Nishimori's condition[10], where no replica symmetry breaking effects are expected. 
This is equivalent to having the correct prior within the Bayesian framework[6] and 
enables one to obtain analytic expressions for some observables as long as some 
gauge requirements are obeyed[10]. Numerical solutions show the emergence of 
stable dominant delta peaks, consistent with those of (5) and (6). The question 
of longitudinal mode stability (corresponding to the replica symmetric solution) 
was addressed by setting initial conditions for the numerical solutions close to the 
solutions (5) and (6), showing that they converge back to these solutions which are 
therefore stable. 
The most interesting quantity to examine is the maximal code rate, for a given 
corruption process, for which messages can be perfectly retrieved. This is defined 
in the case of K, L >_ 3 by the value of R- K/C = N/M for which the free energy of 
the ferromagnetic solution becomes smaller than that of the paramagnetic solution, 
constituting a first order phase transition. A schematic description of the solutions 
obtained is shown in the inset of Fig.la. The paramagnetic solution (m - 0) has 
a lower free energy than the ferromagnetic one (low/high free energies are denoted 
276 Y. Kabashima, T. Murayama, D. $aad and R. Vicente 
by the thick and thin lines respectively, there are no axis lines at m- 0, 1) for 
noise levels p > Pc and vice versa for p_< Pc; both solutions are stable. The critical 
code rate is derived by equating the ferromagnetic and paramagnetic free energies 
to obtain Rc = 1-H2(p) - l+(plog2p+(1 -p)log(1 -p)) This coincides with 
Shannon's capacity. To validate these results we obtained TAP solutions for the 
unbiased message case (K=L=3, C=6) as shown in Fig. la (as +) in comparison 
to Shannon's capacity (solid line). 
Analytical solutions for the saddle point equations cannot be obtained for biased 
patterns and we therefore resort to numerical methods arid the TAP approach. The 
maximal information rate (i.e., code-rate xH(f = (1 + tanhFs)/2) - the source 
redundancy) obtained by the TAP method (O) and numerical solutions of the saddle 
point equations ([3), for each noise level, are shown in Fig. la. Numerical results 
have been obtained using 10 3-10 4 bin models for each probability distribution and 
had been run for 10 5 steps per noise level point. The various results are highly 
consistent and practically saturate Shannon's bound for the same noise level. 
The MN code for K, L > 3 seems to offer optimal performance. However, the main 
drawback is rooted in the co-existence of the stable m = 1 and m - 0 solutions, 
shown in Fig.la (inset), which implies that from some initial conditions the system 
will converge to the undesired paramagnetic solution. Moreover, studying the fer- 
romagnetic solution numerically shows a highly limited basin of attraction, which 
becomes smaller as K and L increase, while the paramagnetic solution at m -- 0 
always enjoys a wide basin of attraction. Computer simulations (see also [3]) show 
that as initial conditions for the decoding process are typically of close-to-zero mag- 
netization (almost no prior information about the original message is assumed) it 
is likely that the decoding process will converge to the paramagnetic solution. 
While all codes with K, L _ 3 saturate Shannon's bound in their equilibrium prop- 
erties and are characterized by a first order, paramagnetic to ferromagnetic, phase 
transition, codes with K = L = 2 show lower performance and different physical char- 
acteristics. The analytical solutions (5) and (6) are unstable at some flip rate levels 
and one resorts to solving the saddle point equations numerically and to TAP based 
solutions. The picture that emerges is sketched in the inset of Fig. lb: The para- 
magnetic solution dominates the high flip rate regime up to the point Pl (denoted 
as 1 in the inset) in which a stable, ferromagnetic solution, of higher free energy, 
appears (thin lines at m = :t:1). At a lower flip rate value P2 the paramagnetic 
solution becomes unstable (dashed line) and is replaced by two stable sub-optimal 
ferromagnetic (broken symmetry) solutions which appear as a couple of peaks in 
the various probability distributions; typically, these have a lower free energy than 
the ferromagnetic solution until P3, after which the ferromagnetic solution becomes 
dominant. Still, only once the sub-optimal ferromagnetic solutions disappear, at the 
spinodal point Ps, a unique ferromagnetic solution emerges as a single delta peak in 
the numerical results (plus a mirror solution). The point in which the sub-optimal 
ferromagnetic solutions disappear constitutes the maximal practical flip rate for the 
current code-rate and was defined numerically (O) and via TAP solutions (-[-) as 
shown in Fig. lb. 
Notice that initial conditions for TAP and the numerical solutions were chosen al- 
most randomly, with a slight bias of O(10-2), in the initial magnetization. The 
TAP dynamical equations are identical to those used for practical BP decoding[5], 
and therefore provide equivalent results to computer simulations with the same pa- 
rameterization, supporting the analytical results. The excellent convergence results 
obtained point out the existence of a unique pair of global solutions to which the 
system converges (below Ps) from practically all initial conditions. This observation 
and the practical implications of using K--L- 2 code have not been obtained by 
Regular and Irregular Gallager-type Error-Correcting Codes 2 77 
information theory methods (e.g.[3]); these prove the existence of very good codes 
for C- L _> 3, and examine decoding properties only via numerical simulations. 
4 Irregular Constructions 
Irregular codes with non-uniform number of non-zero elements per column and 
uniform number of elements per row were recently introduced [11, 12] and were 
found to outperform regular codes. It is relatively straightforward to adapt our 
methods to study these particular constructions. The restriction of the number 
of connections per index can be replaced by a set of N restrictions of the form 
(1), enforcing Ci non-zero elements in the j-th column of the matrix A, and other 
M restrictions enforcing Lt non-zero elements in the/-th column of the matrix B. 
By construction these restrictions must obey the relations Y7= Ci = MK and 
-tM__ Lt = ML. One can assume that a particular set of restrictions is generated 
independently by the probability distributions P(C) and P(L). With that we can 
compute average properties of irregularly constructed codes generated by arbitrary 
distributions. 
Proceeding along the same lines to those of the regular case one can find a very 
similar expression for the free energy which can be interpreted as a mixture of regular 
codes with column weights sampled with probabilities P(C) and P(L). As long as 
we choose probability distributions which vanish for C, L - 0 (avoiding trivial 
non-invertible matrices) and C, L - I (avoiding single checked bits), the solutions 
to the saddle point equations are the same as those obtained in the regular case 
(Eqs.5, 6) leading to exactly the same predictions for the maximum performance. 
The differences between regular and irregular codes show up in their dynamical 
behavior. In the irregular case with K > 2 and for biased messages the basin of 
attraction is larger for higher noise levels [13]. 
5 Conclusion 
In this paper we examined the typical performance of Gallager-type codes. We dis- 
covered that for a certain choice of parameters, either K >_ 3 or L _> 3, one potentially 
obtains optimal performance, saturating Shannon's bound. This comes at the ex- 
pense of a decreasing basin of attraction making the decoding process increasingly 
impractical. Another code, K--L = 2, shows'close to optimal performance with 
a very large basin of attraction, making it highly attractive for practical purposes. 
The decoding performance of both code types was examined by employing the TAP 
approach, an iterative method identical to the commonly used BP. Both numerical 
and TAP solutions agree with the theoretical results. The equilibrium properties of 
regular and irregular constructions is shown to be the same. The improved perfor- 
mance of irregular codes reported in the literature can be explained as consequence 
of dynamical properties. This study examines the typical performance of these in- 
creasingly important error-correcting codes, from which optimal parameter choices 
can be derived, complementing the bounds and empirical results provided in the 
information theory literature . Important aspects that are yet to be investigated 
include other noise types, finite size effects and the decoding dynamics itself. 
Acknowledgement Support by the JSPS RFTF program (YK), The Royal Society and 
EPSRC grant GR/N00562 (DS) is acknowledged. 
278 Y. Kabashima, T. Murayama, D. Saad and R. Vicente 
0.8 
--06 
rr 0.4 
0.2 
0 
(a) Ferro 
+1 ; 
m i 
[ Para 
, ,1:) 
0.1 0.2 0.3 0.4 
p 
).5 
0.8 
0.6 
0.4 
0.2 
0 
b) Ferro 
+1 i ! .. ..... 
'x r .... 'i. '! Para 
,' , I  
0 0.1 0.2 0.3 0.4 0.5 
p 
Figure 1: Critical code rate as a function of the flip rate p, obtained from numerical 
solutions and the TAP approach (N = 104), and averaged over 10 different initial 
conditions with error bars much smaller than the symbols size. (a) Numerical 
solutions for K = L = 3, C = 6 and varying input bias fs ([2) and TAP solutions for 
both unbiased (+) and biased (O) messages; initial conditions were chosen close to 
the analytical ones. The critical rate is multiplied by the source information content 
to obtain the maximal information transmission rate, which clearly does not go 
beyond R= 3/6 in the case of biased messages; for unbiased patterns H2(fs)= 1. 
Inset: The ferromagnetic and paramagnetic solutions as functions of p; thick and 
thin lines denote stable solutions of lower and higher free energies respectively. (b) 
For the unbiased case of K = L = 2; initial conditions for the TAP (+) and the 
numerical solutions (O) are of almost zero magnetization. Inset: The ferromagnetic 
(optimal/sub-optimal) and paramagnetic solutions as functions of p; thick and thin 
lines are as in (a), dashed lines correspond to unstable solutions. 
References 
[1] 
[2] 
[3] 
[4] 
[6] 
[7] 
[8] 
[9] 
[10] 
[11] 
[12] 
C.E. Shannon, Bell $ys. Tech. J., 27, 379 (1948); 27, 623 (1948). 
R.G. Gallager, IRE Trans. Info. Theory, IT-8, 21 (1962). 
D.J.C. MacKay, IEEE Trans. IT, 45,399 (1999). 
D. Thouless, P.W. Anderson and R.G. Palmer, Phil. Mag., 35,593 (1977). 
Y. Kabashima and D. Saad, Europhys. Lett., 44 668 (1998) and 45 97 (1999). 
N. Sourlas, Nature, 339, 693 (1989) and Euro. Phys. Lett., 25, 159 (1994). 
Y. Kabashima, T. Murayama and D. Saad, Phys. Rev. Lett., (1999) in press. 
K.Y.M. Wong and D. Sherrington, J. Phys. A, 20, L793 (1987). 
C. De Dominicis and P. Mottishaw, J. Phys. A, 20, L1267 (1987). 
H. Nishimori, Prog. Theo. Phys., 66, 1169 (1981). 
M. Luby et. al, IEEE proceedings of ISIT98 (1998) and Analysis of Low Density 
Codes and Improved Designs Using Irregular Graphs, preprint. 
D.J.C. MacKay et. al, IEEE Trans. Comm., 47, 1449 (1999). 
R. Vicente et. al, http://xxx.lanl.gov/abs/cond-mat/9908358 (1999). 
