814 
NEU1OMO1PHIC NETWORKS BASED 
ON SPARSE OPTICAL ORTHOGONAL CODES 
Mario P. Vecchi and Jawad A. Salehl 
Bell Communications Research 
435 South Street 
Morristown, NJ 07960-1961 
Abstract 
A family of neuromorphic networks specifically designed for communications 
and optical signal processing applications is presented. The information is encoded 
utilizing sparse Optical Orthogonal Code sequences on the basis of unipolar, binary 
(0, 1) signals. The generalized synaptic connectivity matrix is also unipolar, and 
clipped to binary (0, 1) values. In addition to high-capacity associative memory, 
the resulting neural networks can be used to implement general functions, such as 
code filtering, code mapping, code joining, code shifting and code projecting. 
1 Introduction 
Synthetic neural nets [1,2] represent an active and growing research field. Fundamental 
issues, as well as practical implementations with electronic and optical devices are being 
studied. In addition, several learning algorithms have been studied, for example stochas- 
tically adaptive systems[ 3] based on many-body physics optimization concepts[ 4,5]. 
Signal processing in the optical domain has also been an active field of research. 
A wide variety of non-linear all-optical devices are being studied, directed towards ap- 
plications both in optical computating and in optical switching. In particular, the 
development of Optical Orthogonal Codes (OOC) [6] is specifically interesting to opti- 
c01 communications applications, as it has been demonstrated in the context of Code 
Division Multiple Access (CDMA)[ 7]. 
In this paper we present a new class of neuromorphic networks, specifically designed 
for optical signal processing and communications, that encode the information in sparse 
OOC's. In Section 2 we review some basic concepts. The new neuromorphic networks 
are defined in Section 3, and their associative memory properties are presented in Section 
4. In Section 5 other general network functions are discussed. Concluding remarks are 
given in Section 6. 
2 Neural Networks and Optical Orthogonal Codes 
2.1 Neural Network Model 
Neural network are generally based on multiply-threshold-feedback cycles. In the Hop- 
field model[ 2], for instance, a connectivity  matrix stores the M different memory 
elements, labeled m, by the sum of outer products, 
M 
i,J: 
American Institute of Physics 1988 
815 
where the state vectors _u TM represent the memory elements in the bipolar (-1, 1) basis. 
The diagonal matrix elements in the Hopfield model are set to zero, Til - 0. 
For a typical memory recall cycle, an input vector _v i', which is close to a particular 
memory element rn = k, multiplies the  matrix, such that the output vector v "t is 
given by 
N 
i,j = (2) 
j=l 
and can be seen to reduce to 
(N - + v/(N- 1)(M- 1) 
(3) 
for large N and in the case of randomly coded memory elements u_ TM. 
In the Hopfield model, each output o,t is passed through a thresholding stage 
around zero. The thresholded output signals are then fed back, and the multiply and 
threshold cycle is repeated until a final stable output v_ "t is obtained. If the input v__ i' is 
sufficiently close to u?, and the number of state vectors is small (i.e. M << N), the final 
output will converge to memory element m = k, that is, _v "t  _u k. The associative 
memory property of the network is thus established. 
2.2 Optical Orthogonal Codes 
The OOC sequences have been developed [6'7] for optical CDMA systems. Their prop- 
erties have been specifically designed for this purpose, based on the following two con- 
ditions: each sequence can be easily distinguished from a shifted version of itself, and 
each sequence can be easily distinguished from any other shifted or unshifted sequence 
in the set. Mathematically, the above two conditions are expressed in terms of auto- 
and crosscorrelation functions. Because of the non-negative nature of optical signals , 
OOC are based on unipolar (0, 1) signals [7]. 
In general, a family of OOC is defined by the following parameters: 
- F, the length of the code, 
- K, the weight of the code, that is, the number of 1's in the sequence, 
- ), the auto-correlation value for all possible shifts, other than the zero shift, 
- X, the cross-correlation value for all possible shifts, including the zero shift. 
For a given code length F, the maximum number of distinct sequences in a family 
of OOC depends on the chosen parameters, that is, the weight of the code K and the 
allowed overlap and. In this paper we will consider OOC belonging to the minimum 
overlap class,  = . = 1. 
XWe refer to optical intensity signals, and not to detection systems sensitive to phase information. 
816 
3 Neuromorphic Optical Networks 
Our neuromorphic networks are designed to take full advantage of the properties of the 
OOC. The connectivity matrix T is defined as a sum of outer products, by analogy with 
(1), but with the following important modifications: 
The memory vectors are defined by the sequences of a given family of OOC, with a 
basis given by the unipolar, binary pair (0, 1). The dimension of the sparse vectors 
is given by the length of the code F, and the maximum number of available items 
depends on the chosen family of OOC. 
All of the matrix elements Tij are clipped to unipolar, binary (0, 1) values, resulting 
in a sparse and simplified connectivity matrix, without any loss in the functional 
properties defined by our neuromorphic networks. 
3. The diagonal matrix elements Tii are not set to zero, as they reflect important 
information implicit in the OOC sequences. 
4. The threshold value is not zero, but it is chosen to be equal to K, the weight of 
the OOC. 
The connectivity matrix T is generalized to allow for the possibility of a variety 
of outer product options: self-outer products, as in (1), for associative memory, 
but also cross-outer products of different forms to implement various other system 
functions. 
A simplified schematic diagram of a possible optical neuromorphic processor is shown 
in Figure 1. This implementation is equivalent to an incoherent optical matrix-vector 
multiplier [8], with the addition of nonlinear functions. The input vector is clipped using 
an optical hard-limiter with a threshold setting at 1, and then it is anamorphically 
imaged onto the connectivity mask for . In this way, the i h pixel of the input vector 
is imaged onto the i h colunto of the  mask. The light passing through the mask is 
then anamorphically imaged onto a line of optical threshold elements with a threshold 
setting equal to K, such that the jh row is imaged onto the jh threshold element. 
4 Associative Memory 
The associative memory function is defined by a connectivity matrix MEM given by: 
ij =  zz ; i,j - 1,2...F 
(4) 
where each memory element z__ ' corresponds to a given sequence of the OOC family, 
with code length F. The matrix elements of TMEM are all clipped, unipolar values, as 
indicated by the function 6(}, such that, 
(5) 
817 
We will now show that an input vector z_ k, which corresponds to memory element 
m = k, will produce a stable output (equal to the wanted memory vector) in a single 
pass of the multiply and threshold process. 
The multiplication can be written as: 
F 
= j zj 
We remember that the non-linear clipping function {} is to be applied first to obtain 
MEM. Hence, 
(7) 
k = 0, only the second term in (7) contributes, and the pseudo-orthogonality 
For zi 
properties of the OOC allow us to write: 
zj Oiz..  i 
where the cross-correlation value is Ac < K. 
k 1, we again consider the properties of the OOC to obtain for the first term 
For z i = 
of (7): 
F 
Z k k k=Kz 
(9) 
J 
where K is the weight of the O OC. 
Therefore, the result of the multiplication operation given by (7) can be written as: 
o,,t Kz+[valuestrictly] 
i = less than K 
(lO) 
The thresholding operation follows, around the value K as explained in Section 3. 
That is, (10) is thresholded such that: 
v?t _ -- { I if O?t _ K (11) 
0 if O  < K, 
hence, the final output at the end of a single pass will be given by: v t k 
-- gi' 
The result just obtained can be extended to demonstrate the single pass convergence 
when the input vector is close, but not necessarily equal, to a stored memory element. 
We can draw the following conclusions regarding the properties of our neuromorphic 
networks based on OOC: 
 For any given input vector v_ i', the single pass output will correspond to the 
memory vector z__ " which has the smallest Hamming distance to the input. 
 If the input vector v_ i' is missing a single 1-element from the K 1's of an OOC, 
the single pass output will be the null or zero vector. 
818 
 If the input vector v_/n has the same Hamming distance to two (or more) memory 
vectors z_ ", the single pass output will be the logical sum of those memory vectors. 
The ideas just discussed were tested with a computer simulation. An example of 
associative memory is shown in Table 1, corresponding to the OOC class of length 
F = 21 and weight K = 2. For this case, the maximum number of independent 
sequences is M = 10. The connectivity matrix MEM is seen in Table 1, where one can 
clearly appreciate the simplifying features of our model, both in terms of the sparsity 
and of the unipolar, clipped values of the matrix elements. The computer simulations for 
this example are shown in Table 2. The input vectors a and b show the error-correcting 
memory recovery properties. The input vector c is equally distant to memory vectors 
z s and z_ s, resulting in an output which is the sum (z_ s ( z_s). And finally, input vector 
d is closest to z  but one I is missing, and the output is the zero vector. The mask 
in Figure I shows the optical realization of the Table 1, where the transparent pixels 
correspond to the 1's and the opaque pixels to the O's of the connectivity matrix ME. 
It should be pointed out that the capacity of our network is significant. From the 
previous example, the capacity is seen to be  F/2 for single pass memory recovery. 
This result compares favorably with the capacity of a Hopfield model [9], of  F/4 In F. 
5 General Network Functions 
Our neuromorphic networks, based on OOC, can be generalized to perform functions 
other than associative memory storage by constructing non-symmetrical connectivity 
matrices. The single pass convergence of our networks avoids the possibility of lhnit- 
cycle oscillations. We can write in general: 
= V , 
where each pair defined by m includes two vectors y" and z_ m, which are not necessarily 
equal. The clipping function (} insures that all matrix elements are binary (0,1) values. 
The possible choice of vector pairs is not completely arbitrary, but there is a wide variety 
of functions that can be implemented for each family of OOC. We will now discuss some 
of the applications that are of particular interest in optical communication systems. 
5.1 Code Filtering (CDMA) 
Figure 2 shows an optical CDMA network in a star configuration. M nodes are inter- 
connected with optical fibers to a passive MxM star coupler that broadcasts the optical 
signals. At each node there is a data encoder that maps each bit of information to the 
OOC sequence corresponding to the user for which the transmission is intended. In 
addition, each node has a filter and decoder that recognizes its specific OOC sequence. 
The optical transmission rate has been expanded by a factor F corresponding to the 
length of the OOC sequence. Within the context of a CDMA communication system [7], 
the filter or decoder must perform the function of recognizing a specific OOC sequence 
in the presence of other interfering codes sent on the common transmission medium. 
819 
We can think, then, of one of our neuromorphic networks as a filter, placed at a given 
receiver node, that will recognize the specific code that it was programmed for. 
We define for this purpose a connectivity matrix as 
CDMA k  . i,j = 1,2...F, (13) 
'ij -- Z i Zj , 
where only one vector z_ k is stored at each node. This symmetric, clipped connectivity 
matrix will give an output equal to z? whenever the input contains this vector, and a 
null or zero output vector otherwise. It is clear by comparing (13) with (4) that the 
CDMA filtering matrix is equivalent to an associative memory matrix with only one 
item imprinted in the memory. Hence the discussion of Section 4 directly applies to the 
understanding of the behaviour of TCDMA. 
In order to evaluate the performance of our neuromorphic network as a CDMA 
filter, computer simulations were performed. Table 3 presents the CDMA matrix for 
a particular node defined by _z k of a CDMA system based on the OOC family F = 21, 
K = 2. The total number of distinct codes for this OOC family is M = 10, hence there 
are 9 additional OOC sequences that interfere with z k, labeled in Table 3 z_  to z 9. 
The performance was simulated by generating random composite sequences from the 
set of codes z  to z__ 9 arbitrarily shifted. All inputs are unipolar and clipped (0,1) signals. 
The results presented in Table 4 give examples of our simulation for the TCDMA matrix 
shown in Table 3. The input a is the (logical) sum of a 1-bit (vector zk), plus interfering 
signals from arbitrarily shifted sequences of z_ z, z_ a, z__ 4, z_ a and z_ 9. The output of the 
neuromorphic network is seen to recover accurately the desired vector z?. The input 
vector _b contains a 0-bit (null vector), plus the shifted sequences of z_  , z_ z, z_ a, z_ a, z ? 
and z_ a, and we see that the output correctly recovers a O-bit. 
As discussed in Section 4, our neuromorphic network will always correctly recognize 
a 1-bit (vector z?) presented to its input. On the other hand 2, there is the possibility of 
making an error when a 0-bit is sent, and the interfering signals from other nodes happen 
to generate the chip positions of z?. This case is shown by input vector c of Table 4, 
which contains a 0-bit (null vector), plus shifted sequences of z_ z, z_ a, z_ 4, z_ s, z_ a, z_ ? and 
z__ a in such a way that the output is erroneously given as a 1-bit. The properties of the 
OOC sequences are specifically chosen to minimize these errors[ 7], and the statistical 
results of our simulation are also shown in Table 4. It is seen that, as expected, when 
a 1-bit is sent it is always correctly recognized. On the other hand, when 0-bits are 
sent, occasional errors occur. Our simulation, yields an overall bit error rate (BER) of 
BER,im = 5.88%, as shown in Table 4. 
These results can be compared with theoretical calculations [7] which yield an esti- 
mate for the BER for the CDMA system described: 
B E Rc,ic   k=0 
(14) 
where q =_ 1 - K For the example of the OOC family F = 21, K = 2, with M = 10, 
2F' 
the above expression yields BERc,I  5.74%. 
2Our channel can be described, then, as a binary Z-channel between each two nodes dynamically 
establishing a communication path 
82O 
It is seen, therefore, that our neuromorphic network approaches the minimum pos- 
 . --GDMA 
sible BER for a given family of OOC. In fact, the results obtmned usm our T 
are equivalent CDMA detection scheme based on "optical-AND-gates "[10], which cor- 
responds to the limiting BER determined by the properties of the OOC themselves s. 
The optical mask corresponding to the code filtering function is shown in Figure 3. 
5.2 Other Functions 
As a first example of a non-symmetric T matrix, let us consider the function of mapping 
an input code to a corresponding different output code. We define our mapping matrix 
as; 
Yi zj 
where an input vector z_ TM will produce a different output vector code y_m. 
The function of code joining is defined by a transfer function that takes a given 
input code and produces at the output a chosen combination of two or more codes. 
This function is performed by expressing the general matrix given by 12 as follows: 
where an input vector z_ TM will result in an output that joins several vector codes (_ym  
The code shifting matrix $HIFT will allow for the shift of a given code sequence, 
such that both input and output correspond to the same code, but shifted with respect 
to itself. That is, 
ij = G .z(s)z(O) ; i,j = 1,2...F, (17) 
m 
where we have indicated an unshifted code sequence by z_(0) TM, and its corresponding 
output pair as a shifted version of itself z__(s) ". 
The code projecting function corresponds to processing an input vector that contains 
the logical sum of several codes, and projecting at the output a selected single code 
sequence. The corresponding matrix PRoJ is given by: 
08) 
where each input vector (ym (t) w_ TM (t) ...) will project at the output to a single code 
_z TM. In general, the resulting output code sequence z_ TM could correspond to a code not 
necessarely contained in the input vector. 
The performance and error correcting properties of these, and other, general func- 
tions follow a similar behaviour as discussed in Section 4. 
3The BER for the OOC family shown in this example are far too large for a useful CDMA com- 
munications system. Our choice intended to show computer simulated results within a reasonable 
computation time. 
821 
6 Conclusions 
The neuromorphic networks presented, base d on sparse Optical Orthogonal Code (O O C) 
sequences, have been shown to have a number of attractive properties. The unipolar, 
clipped nature of the synaptic connectivity matrix simplifies the implementation. The 
single pass convergence further allows for general network functions that are expected 
to be of particular interest in communications and signal processing systems. 
The coding of the information, based on OOC, has also been shown to result in high 
capacity associative memories. The combination of efficient associative memory prop- 
erties, plus a variety of general network functions, also suggests the possible application 
of our neuromorphic networks in the implementation of computational functions based 
on optical symbolic substitution. 
The family of neuromorphic networks discussed here emphasizes the importance of 
understanding the general properties of non-negative systems based on sparse codes [11]. 
It is hoped that our results will stimulate further work on the fundamental relationship 
between coding, or representations, and the information processing properties of neural 
nets. 
Acknowledgement 
We thank J. Y. N. Hui and J. Alspector for many useful discussions, and C. A. Brackett for his support 
and encouragement of this research. 
References 
[1] S. Grossberg. In K. Schmitt, editor, Delay and Functional-Differential Equations and Their Ap- 
plications, page 121, Academic Press, New York, NY, 1972. 
[2] J. J. Hopfield. Neural Networks and Physical Systems with Emergent Collective Computational 
Abilities. Proc. Nat. Acad. Sci. USA, 79:2254, 1982. 
[3] D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. A Learning Algorithm for Boltzmann Machines. 
Cogn. Sci., 9:147, 1985. 
[4] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by Simulated Annealing. Science, 
220:671, 1983. 
[5] M.P. Vecchi and S. Kirkpatrick. Global Wiring by Simulated Annealing. IEEE Trans. CAD of 
Integrated Circuits and Systems, CAD-2:215, 1983. 
[6] F. R. K. Chung, J. A. Salehi, and V. K. Wei. Optical Orthogonal Codes: Design, Analysis and 
Applications. In IEEE International Symposium on Information Theory, Catalog No. 86CH374-7, 
1986. Accepted for publication in IEEE Trans. on Information Theory. 
[7] J. A. Salehi and C. A. Brackett. Fundamental Principles of Fiber Optics Code Division Multiple 
Access. In IEEE International Conference on Communications, 1987. 
[8] N.H. Farhat, D. Psaltis, A. Prata, and E. Paek. Optical Implementation of the Hopfield Model. 
Appl. Opt., 24:1469, 1985. 
[9] R. J. McEliece, E. C. Posner, E. R. Rodemich, and S.S. Venkatesh. The Capacity of Hopfield 
Associative Memory. IEEE Trans. on Information Theory, IT-33:461, 1987. 
[10] J. A. Salehi. Principles and Applications of Optical AND Gates in Fiber Optics Code Division 
Multiple Access Networks. In preparation, 1987. 
[11] G. Palm. Technical comments. Science, 235:1226, 1987. 
822 
Tsbb I: Assoc,mfi,N Mery. F, ximpl sbo,iq d'i&, d' 10 distinct 
iu O O i O I O O O O O O O O O O O O O O O O 
1 O O O I O O I O O O O O O O O O O O O O O 
t O O O O O I O O O I O O O O O O O  0 O O 
_t* O O O O O O O I O O O O I O O O O O 0 O O 
i" O O O O O O O O I O O O O O I O O'b' O O O 
i O O O O O O O O O O I O O O O O O I O O O 
is iII I I I I I I I I I I I I i i i I i I I 
i I O O I O I O O O O O O O I O O O 0 O O 
i i O O O O O I I O I O O O O O I O I I O 
O O I O i O O O O O O O O O O O O O O O O' 
O O O I O O i O O O O O O O O O O O O O O 
O O i O I O O O O O O O O i O O O I O O O 
000001000100000100000 
O O O I O O i O O O O O O O 0 O O O O O O 
O O O I O O O I O O O O I O O O O O O O O 
O O I I O O O O I I I I I O i O I I I O O 
I O I I O I O O I I I I O O O O O O 0 O O 
O O O O O O O O O O I O O O O O O I O O O 
O O O O O O O I O O O I O O O I O I 0 I I 
O O O O O O O i O O O O i O O O O O O O O 
O 0 O O I O O O O O O O O i O O O O O O O 
I I 0 I I I I I I I I I I I I I I I i 0 I 
O O O O O I O O O O O O O O O i O O O 0 O 
I I I I I I O I I O 0 I I O I O I I 0 O 
I O O I I I O O O O I I I O O 0 O I O I I 
O O O O O O O I I I I O O O I I I I I I O 
O I O O O O O O O O I O O I O O I I i I 
O O I O O O O O O O I O 0 I O O O I O I OJ 
Code Faterls s 
I OOC Imil,  = it, K = 3 j 
i'je o o o e o e I o o o o o t o o e o e e 
lllllllllllllllllllOl 
llllllllllllllllllllO 
lllllllllllllllllllll 
O I I O O I O O O I I I O I I I O O I I I 
O I I O O O O I O I I O O I I O O I O I 0 
O I I I I I O O O 0 O O O I I O O O O O I 
O O O O O O O I O I I O O O O O O O I I O 
O O O I O O O 0 O I O I O I I O O O I 00 
O O O O O O O O I O O O O O I O O 0 O O O 
O O I 0 I O O O I I I O O O O O I O I I O 
O O O I O O O I I I 0 O I O I O O I 0 I 0 
I I O O O O I O I O O I O O I I O I O I I 
I I I I O O I O I I O O O O O O I I I 0 I 
O I O I O I I O I I I O O I I O I 0 I I O 
O I O O O O I O I I O O O O I O O O I I I 
I I  I O 0 I O O I O O I I I I I I I O I 
I I 0 I O O I O O I O O O O I O I O O I I 
O O I I O O I O I O O O O I I O I I O I O 
O O 0 O O O O O O O O O O O I O O I O I O 
I O I O O 0 O O O O I O O I I O O I O 0 I 
O O I O O O O O I O O I O I 0 O I I O O I 
f] I I I I I I I I I I I I I I I I I I I 0 l 
Is I t I 0 I I I I I I I I I I I I I I I I I 
 O O O O O O O I O O I O O O O O O I O O O 
fflllllllllllllllllllOI 
 O O O O O I O O O O O O O O O I 0 O O O O 
I '"'" v""" 
I l' III o o o o o o o o o o 
, Outpsd Vc14 
OutPut Vlcted mmw 
i  ]l I O O O I I O t I O O I I I O 0 O O 
I SJt bt J B elMvlom, 
StOI 
11.1'1% st8% 'r 
823 
Figure 1: 
Schematic diagram of an optical neuromorphic pro- 
cessor using sparse Optical Orthogonal Codes. No- 
tice the absence of teedback because of the single-pass 
convergence. The mask shown represents the realiza- 
tion of the content-addressable memory of Table 1. 
Figure 3: 
Optical realization of a code filtering (CDMA) mask 
of Table 3. The l's are represented by the transpar. 
ent pixels, and the 0s by the opaque pixels. 
Figure 2: 
Schematic diagram of a CDMA communications sys- 
tem over an Optical Fiber interconnection network. 
Each node represents one of the M possible distinct 
users in the system. 
