VLSI Implementation of a High-Capacity Neural Network 793 
VLSI Implementation of a High-Capacity 
Neural Network Associative Memory 
Tzi-Dar Chiueh  and Rodney M. Goodman 
Department of Electrical Engineering (116-81) 
California Institute of Technology 
Pasadena, CA 91125, USA 
ABSTRACT 
In this paper we describe the VLSI design and testing of a high 
capacity associative memory which we call the exponential cor- 
relation associative memory (ECAM). The prototype 3p-CMOS 
programmable chip is capable of storing 32 memory patterns of 
24 bits each. The high capacity of the ECAM is partly due to the 
use of special exponentiation neurons, which are implemented via 
sub-threshold MOS transistors in this design. The prototype chip 
is capable of performing one associative recall in 3 
I ARCHITECTURE 
Previously (Chiueh, 1989), we have proposed a general model for correlation-based 
associative memories, which includes a variant of the Hopfield memory and high- 
order correlation memories as special cases. This new exponential correlation as- 
sociative memory (ECAM) possesses a very large storage capacity, which scales 
eaponentially with the length of memory patterns (Chiueh, 1988). Furthermore, it 
has been shown that the ECAM is asymptotically stable in both synchronous and 
I Tzi-Dar Chlueh is now with the Department of Electrical Engineering, National Taiwan Uni- 
versity, Taipei, Taiwan 10764. 
794 Chiueh and Goodman 
asynchronous updating modes (Chiueh, 1989). The model is based on an archi- 
tecture consisting of binary connection weights, simple hard-limiter neurons, and 
specialized nonlinear circuits as shown in Figure 1. The evolution equation of this 
general model is 
x' - sgn -/(<u , x>) u (k) , (1) 
k--1 
where u(1), u(2), .. ., u(M) are the M memory patterns. x and x  are the current and 
the next state patterns of the system respectively, and sgn is the threshold function, 
which takes on the value +1 if its argument is nonnegative, and -1 otherwise. 
We addressed, in particular, the case where f(.) is in the form of an exponentiation, 
namely, when the evolution equation is given by 
x' --- sgn a<U(k),x> u() , (2) 
and a is a constant greater than unity. 
The ECAM chip we have designed is programmable; that is, one can change the 
stored memory patterns at will. To perform an associative recall, one first loads a 
set of memory patterns into the chip. The chip is then switched to the associative 
recall mode, an input pattern is presented to the ECAM chip, and the ECAM chip 
then computes the next state pattern according to Equation (2). The components 
of the next state pattern appear at the output in parallel after the internal circuits 
have settled. Feedback is easily incorporated by connecting the output port to the 
input port, in which case the chip will cycle until a fixed point is reached. 
2 DESIGN OF THE ECAM CIRCUITS 
From the evolution equation of the ECAM, we notice that there are essentially three 
circuits that need to be designed in order to build an ECAM chip. They are: 
< u , x >, the correlation computation circuit; 
M 
a <u(k)' x> u (), the exponentiation, multiplication and summing circuit; 
sgn(.), the threshold circuit. 
We now describe each circuit, present its design, and finally integrate all these 
circuits to get the complete design of the ECAM chip. 
VLSI Implementation of a High-Capacity Neural Network 795 
2.1 CORRELATION COMPUTATION 
In Figure 2, we illustrate a voltage-divider type circuit consisting of NMOS transis- 
tors working as controlled resistors (linear resistors or open circuits). This circuit 
computes the correlation between the input pattern x and a memory pattern u (k). 
If the ith components of these two patterns are the same, the corresponding XoR 
gate outputs a "0" and there is a connection from the node V(ukx ) to VBB; other- 
wise, there is a connection from V(ukx ) to GND. Hence the output voltage will be 
proportional to the number of positions at which x and u  match. The maximum 
output voltage is controlled by an externally supplied bias voltage Vs. Normally, 
V is set to a voltage lower than the threshold voltage of NMOS transistors (VTH) 
for a reason that will be explained later. Note that the conductance of an NMOS 
transistor in the ON mode is not fixed, but rather depends on its gate-to-source 
voltage and its drain-to-source voltage. Thus, some nonlinearity is bound to occur 
in the correlation computation circuit, however, simulation shows that this effect is 
small. 
2.2 EXPONENTIATION, MULTIPLICATION, AND SUMMATION 
Figure 4 shows a circuit that computes the exponentiation of V  the product of 
ux , 
the u? ) and the exponential, and the sum of all M products. 
The exponentiation function is implemented by an NMOS transistor whose gate 
voltage is V(ukx ). Since VSB, the maximum value that V(ukx ) can assume, is set to be 
lower than the threshold voltage (VTH); the NMOS transistor is in the subthreshold 
region, where its drain current depends exponentially on its gate-to-source voltage 
(Mead, 1989). If we temporarily ignore the transistors controlled by u? ) or the 
complement of u? ), the current flowing through the exponentiation transistor asso- 
ciated with V(ukx ) will scale exponentially with V(ukx ). Therefore, the exponentiation 
function is properly computed. 
Since the multiplier u? ) assumes either --1 or -1, the multiplication can be easily 
done by forming two branches, each made up of a transmission gate in series with an 
exponentiation transistor whose gate voltage is V(ukx ). One of the two transmission 
gates is controlled by u? ), and the other by the complement of u? ). Consequently, 
when u? ) -- 1, the positive branch will carry a current that scales exponentially 
with the correlation of the input x and the k th memory pattern u , while the 
negative branch is essentially an open circuit, and vice versa. 
Summation of the M terms in the evolution equation is done by current summing. 
The final results are two currents If and I', which need to be compared by a 
threshold circuit to determine the sign of the ith bit of the next state pattern x. 
In the ECAM a simple differential amplifier (Figure 3) performs the comparison. 
796 Chiueh and Goodman 
2.3 THE BASIC ECAM CELL 
The above computational circuits are then combined with a simple static RAM cell, 
to make up a basic ECAM cell as illustrated in Figure 5. The final design of an 
ECAM that stores M N-bit memory patterns can be obtained by replicating the 
basic ECAM cell M times in the horizontal direction and N times in the vertical 
direction, together with read/write circuits, sense amplifiers, address decoders, and 
I/O multiplexers. The prototype ECAM chip is made up of 32 x 24 ECAM cells, 
and stores 32 memory patterns each 24 bits wide. 
3 ECAM CHIP TEST RESULTS 
The test procedure for the ECAM is to first generate 32 memory patterns at random 
and then program the ECAM chip with these 32 patterns. We then pick a memory 
pattern at random, flip a specified number of bits randomly, and feed the resulting 
pattern to the ECAM as an input pattern (x). The output pattern (x ) can then be 
fed back to the inputs of the ECAM chip. This iteration continues until the pattern 
at the input is the same as that at the the output, at which time the ECAM chip 
is said to have reached a stable state. We select 10 sets of 32 memory patterns and 
for each set we run the ECAM chip on 100 trial input patterns with a fixed number 
of errors. Altogether, the test consists of 1000 trials. 
In Figure 6, we illustrate the ECAM chip test results. The number of successes is 
plotted against the number of errors in the input patterns for the following four 
cases: 1) The ECAM chip with Vss = 5V; 2) Vss = 2V; 3) VBs - 1V; and 4) a 
simulated ECAM in which the exponentiation constant a, equals 2. It is apparent 
from Figure 6 that as the number of errors increases, the number of successes 
decreases, which is expected. Also, one notices that the simulated ECAM is by far 
the best one, which is again not unforeseen because the ECAM chip is, after all, 
only an approximation of the ideal ECAM model. 
What is really unexpected is that the best performance occurs for VBs = 2V rather 
than Vss = 1V (VrI-I in this CMOS process). This phenomenon arises because 
of two contradictory effects brought about by increasing Vss. On the one hand, 
increasing VBs increases the dynamic range of the exponentiation transistors in the 
ECAM chip. Suppose that the correlations of two memory patterns u(0 and u  
with the input pattern x are tt and tk, respectively, where tt > t; then 
V(!) _ (t I qt_ N) Vss V(ux) = (t + N) 
x - 2N ' 2N 
Therefore, as VBB increases, so does the difference between V (1) and V(ukx ), and u  
ux 
becomes more dominant than u() in the weighted sum of the evolution equation. 
VLSI Implementation of a High. Capacity Neural Network 797 
Hence, as VBB increases, the error correcting ability of the ECAM chip should 
improve. On the other hand, as VB increases beyond the threshold voltage, the 
exponentiation transistors leave the subthreshold region and may enter saturation, 
where the drain current is approximately proportional to the square of the gate- 
to-source voltage. Since a second-order correlation associative memory in general 
possesses a smaller storage capacity than an ECAM, one would expect that with 
a fixed number of loaded memory patterns, the ECAM should do better than the 
second-order correlation associative memory. Thus one effect tends to enhance the 
performance of the ECAM chip, while the other tends to degrade it. A compromise 
between these two effects is reached, and the best performance is achieved when 
V = 2V. 
For the case when VBB = 2V, the drain current versus gate-to-source voltage char- 
acteristic of the exponentiation transistors is actually a hybrid of a square function 
and an exponentiation function. At the bottom it is of an exponential form, and 
it gradually fiattens out to a square function, once the gate-to-source voltage be- 
comes larger than the threshold voltage. Therefore, the ECAM chip with Vss = 
2V is a mixture of the second-order correlation associative memory and the pure 
ECAM. According to the convergence theorem for correlation associative memories 
(Chiueh, 1989) and the fact that f(.) in the ECAM chip with Vss = 2V is still 
monotonically nondecreasing, the ECAM chip is still asymptotically stable when 
Vss = 2V. 
We have tested the speed of the ECAM chip using binary image vector quantization 
as an example problem. The speed at which the ECAM chip can vector-quantize 
binary images is of interest. We find experimentally that the ECAM chip is capable 
of doing one associative recall operation, in less than 3/s, 'n 4 x 4 blocks. This 
projects to approximately 49 ms for a 512 x 512 binary image, or more than 20 
images per second. 
4 CONCLUSIONS 
In this paper, we have presented a VLSI circuit design for implementing a high 
capacity correlation associative memory. The performance of the ECAM chip is 
shown to be almost as good as a computer-simulated ECAM. Furthermore, we 
believe that the ECAM chip is more robust than an associative memory using a 
winner-take-all function, because it obtains its result via iteration, as opposed to 
one shot. In conclusion, we believe that the ECAM chip provides a fast and efficient 
way for solving many associative recall problems, such as vector quantization and 
optical character recognition. 
Acknowledgement 
This work was supported in part by NSF grant No. MIP - 8711568. 
798 Chiueh and Goodman 
References 
T. D. Chiueh and R. M. Goodman. (1988) "High Capacity Exponential Associative 
Memory," in Proc. of IEEE ICNN, Vol. I, pp. 153-160. 
T. D. Chiueh. (1989) "Pattern Classification and Associative Recall by Neural 
Networks," Ph.D. dissertation, California Institute of Technology. 
C. A. Mead. (1989) Analog VLSI and Neural Systems. Reading, MA: Addison- 
Wesley. 
x T U 
x 2 
Figure 1: Architecture of the General Correlation-Based Associative Memory 
1 VBB 
! 
U 1 X 1 U X 2 U X U X 
N-I N-I N N 
Figure 2: The Correlation Computation Circuit 
VLSI Implementation of a High-Capacity Neural Network 799 
VDD 
? 
I I 
Figure 3: The Threshold Circuit 
(1) 
u i 
-- u(1) 
i 
(2) 
v 
ux 
1 
u? ) 
1 
(M) 
Figure 4: The Exponentiation, Multiplication, and Summation Circuit 
800 Chiueh and Goodman 
x i 'r I i Ii Y 
VBn 
t 
u i 
' 
cell . 
) 
U. 
1 
Figure 5: Circuit Diagram of the Basic ECAM Cell 
1000 
900 
700 
500 
400 
300 
200 
100 
0 
0 
*-- Simulation (a=2) 
*- Vbb = 5V 
" Vbb = 2V  
- Vbb = 1V 
I I I I I I I 
2 3 4 5 6 7 
Number of errors in input patterns 
Figure 6: Error Correcting Ability of the ECAM Chip with Different VBB compared 
with a Simulated ECAM with a = 2 
