7O3 
WINNER-TAKE-ALL 
NETWORKS OF O(N) COMPLEXITY 
J. Lazzaro, S. Ryckebusch, M.A. Mahowald, and C. A. Mead 
California Institute of Technology 
Pasadena, CA 91125 
ABSTRACT 
We have designed, fabricated, and tested a series of compact CMOS 
integrated circuits that realize the winner-take-all function. These 
analog, continuous-time circuits use only O{n) of interconnect to 
perform this function. We have also modified the winner-take-all 
circuit, realizing a circuit that computes local nonlinear inhibition. 
Two general types of inhibition mediate activity in neural systems: subtractive in- 
hibition, which sets a zero level for the computation, and multiplicative (nonlinear) 
inhibition, which regulates the gain of the computation. We report a physical real- 
ization of general nonlinear inhibition in its extreme form, known as winner-take-all. 
We have designed and fabricated a series of compact, completely functional CMOS 
integrated circuits that realize the winner-take-all function, using the full analog 
nature of the medium. This circuit has been used successfully as a component 
in several VLSI sensory systems that perform auditory localization (Lazzaro and 
Mead, in press) and visual stereopsis (Mahowald and Delbruck, 1988). Winner- 
take-all circuits with over 170 inputs function correctly in these sensory systems. 
We have also modified this global winner-take-all circuit, realizing a circuit that 
computes local nonlinear inhibition. The circuit allows multiple winners in the net- 
work, and is well suited for use in systems that represent a feature space topograph- 
ically and that process several features in parallel. We have designed, fabricated, 
and tested a CMOS integrated circuit that computes locally the winner-take-all 
function of spatially ordered input. 
704 Lazzaro, Ryckebusch, Mahowald and Mead 
THE WINNER-TAKE-ALL CIRCUIT 
Figure 1 is a schematic diagram of the winner-take-all circuit. A single wire, asso- 
ciated with the potential Vc, computes the inhibition for the entire circuit; for an 
n neuron circuit, this wire is O(n) long. To compute the global inhibition, each 
neuron k contributes a current onto this common wire, using transistor T2k. To 
apply this global inhibition locally, each neuron responds to the common wire volt- 
age Vc, using transistor Txk. This computation is continuous in time; no clocks 
are used. The circuit exhibits no hysteresis, and operates with a time constant 
related to the size of the largest input. The output representation of the circuit 
is not binary; the winning output encodes the logarithm of its associated input. 
Figure 1. Schematic diagram of the winner-take-all circuit. Each neuron receives 
a unidirectional current input I; the output voltages Vx... V, represent the result 
of the winner-take-all computation. If I = max(Ix... I,), then V is a logarithmic 
function of I; if I i ' I, then Vi  0. 
A static and dynamic analysis of the two-neuron circuit illustrates these system 
properties. Figure 2 shows a schematic diagram of a two-neuron winner-take-all 
circuit. To understand the behavior of the circuit, we first consider the input 
condition Ix - I2 -- In. Transistors Tx a'd Tx have identical potentials at gate 
and source, and are both sinking I,; thus, the drain potentials Vx and V2 must be 
equal. Transistors T2 and T2 have identical source, drain, and gate potentials, 
and therefore must sink the identical current I -- I 2 - I/2. In the subthreshold 
region of operation, the equation Im- Io exp(Vc/Vo) describes transistors Tx and 
Tx, where Io is a fabrication parameter, and Vo - kT/qr. Likewise, the equation 
Ic/2 - Io exp((Vm- Vc)/Vo), where Vm -- V - V2, describes transistors T2 and 
T2. Solving for V, (I,, I) yields 
= Voln() 
Io + Vln(2Io ' 
Winner-Take-All Networks of O(N) Complexity 705 
Thus, for equal input currents, the circuit produces equal output voltages; this 
behavior is desirable for a winner-take-all circuit. In addition, the output voltage 
V, logarithmically encodes the magnitude of the input current I,. 
Figure 2. Schematic diagram of a two-neuron winner-take-all circuit. 
The input condition Ii =Im + 8i, I2 =Im illustrates the inhibitory action of the 
circuit. Transistor Tl must sink 5i more current than in the previous example; as a 
result, the gate voltage of Tx rises. Transistors Tx and Ti2 share a common gate, 
however; thus, Tx must also sink I, + 5i. But only Im is present at the drain of 
T12. To compensate, the drain voltage of Tl, V2, must decrease. For small 5s, the 
Early effect serves to decrease the current through Tx, decreasing V2 linearly with 
i. For large is, Tx must leave saturation, driving V2 to approximately 0 volts. 
As desired, the output associated with the smaller input diminishes. For large 
Ic  O, and Ic  Ic. The equation lm+ 8 = Io exp(V,,/Vo) describes transistor 
Tx,, and the equation = Ioexp((Vx- Vc)/Vo) describes transistor Solving 
for V1 yields 
V1 '-- Voln(Im'i i ) _. Vo ln(/C ) 
Zo ' (2) 
The winning output encodes the logarithm of the associated input. The symmetrical 
circuit topology ensures similar behavior for increases in I2 relative to 
Equation 2 predicts the winning response of the circuit; a more complex expression, 
derived in (Lazzaro et.al., 1989), predicts the losing and crossover response of the 
circuit. Figure 3 is a plot of this analysis, fit to experimental data. Figure 4 shows 
the wide dynamic range and logarithmic properties of the circuit; the experiment in 
Figure 3 is repeated for several values of 12, ranging over four orders of magnitude. 
The conductance of transistors Tlx and Tx2 determines the losing response of the 
circuit. Variants of the winner-take-all circuit shown in (Lazzaro et. al., 1988) 
achieve losing responses wider and narrower than Figure 3, using circuit and mask 
layout techniques. 
706 Lzzaro, Ryckebusch, Mahowald and Mead 
WINNER-TAKE-ALL TIME RESPONSE 
A good winner-take-all circuit should be stable, and should not exhibit damped 
oscillations ('ringing ) in response to input changes. This section explores these 
dynamic properties of our winner-take-all circuit, and predicts the temporal re- 
sponse of the circuit. Figure 8 shows the two-neuron winner-take-all circuit, with 
capacitances added to model dynamic behavior. 
Figure 8. Schematic diagram of a two-neuron winner-take-all circuit, with capac- 
itances added for dynamic analysis. c is a large MOS capacitor added to each 
neuron for smoothing; co models the parasitic capacitance contributed by the gates 
of T and T12, the drains of T21 and T22, and the interconnect. 
(Lazzaro et. al., 1988) shows a small-signal analysis of this circuit. The transfer 
function for the circuit has real poles, and thus the circuit is stable and does not ring, 
if Io > 4/(/c), where I  I2  I. Figure 9 compares this bound with experimental 
data. 
If L > 4I(Co/C), the circuit exhibits first-order behavior. The time constant CVo/l 
sets the dynamics of the winning neuron, where Vo = kT/qr  40 mV. The time 
constant cvz/l sets the dynamics of the losing neuron, where vz  50 v. Figure 10 
compares these predictions with experimental data. 
Winner-Take-All Networks of O(N) Complexity 707 
2.4 
2.2' 
2.0' 
1.8' 
1.6' 
1.4' 
1.2' 
1.0 
0.92 0.94 0.96 0.98 1.00 1.02 1.04 1.06 1.08 
I/I 
Figure $. Experimental data (circles) and theory (solid lines) for a two-neuron 
winner-take-all circuit. I, the input current of the first neuron, is swept about the 
value of I2, the input current of the second neuron; neuron voltage outputs v and 
'2 axe plotted versus normalized input current. 
2.6 
2.4  
2.2 
2.0  
1.8 
1.6  
1.4 
1.2  
1.0 
10-12 10- 10-o 10- 10-s 
Z(A) 
Figure 4. The experiment of Figure 3 is repeated for several values of I2; experi- 
mental data of output voltage response axe plotted versus absolute input current on 
a log scale. The output voltage v = v2 is highlighted with a circle for each experi- 
ment. The dshed line is a theoretical expression confirming logaxithmic behavior 
over four orders of magnitude (Equation 1). 
708 Lazzaro, Ryckebusch, Mahowald and Mead 
10--7, 
10-11 10-o 10-o 10-s 10-? 
I 
Figure 9. Experimental data (circles) and theoretical statements (solid line) for a 
two-neuron winner-take-all circuit, showing the smallest Io, for a given I, necessary 
for a first-order response to a small-signal step input. 
10--1 
10- 
10-a I '   
10_4 I 
10-8     
10-1 10- 10-1o 10- 10-s 10-? 10-e 
Z(A) 
Figure 10. Experimental data (symbols) and theoretical statements (solid line) for 
a two-neuron winner-take-all circuit, showing the time constant of the first-order 
response to a small-signal step input. The winning response (filled circles) and losing 
response (triangles) of a winner-take-all circuit are shown; the time constants differ 
by several orders of magnitude. 
WinnersTake-All Networks of O(N) Complexity 709 
THE LOCAL NONLINEAR INHIBITION CIRCUIT 
The winner-take-all circuit in Figure 1, as previously explained, locates the largest 
input to the circuit. Certain applications require a gentler form of nonlinear inhibi- 
tion. Sometimes, a circuit that can represent multiple intensity scales is necessary. 
Without circuit modification, the winner-take-all circuit in Figure 1 can perform 
this task. (Lazzaro et. al., 1988) explains this mode of operation. 
Other applications require a local winner-take-all computation, with each winner 
having influence over only a limited spatial area. Figure 12 shows a circuit that 
computes the local winner-take-all function. The circuit is identical to the original 
winner-take-all circuit, except that each neuron connects to its nearest neighbors 
with a nonlinear resistor circuit (Mead, in press). Each resistor conducts a current 
Ir in response to a voltage AV across it, where L = lotanh(Av/(2Vo)). Io, the 
saturating current of the resistor, is a controllable parameter. The current source, 
L, present in the original winner-take-all circuit, is distributed between the resistors 
in the local winner-take-all circuit. 
l+x ( 
V+x 
T1+1 
)L 
Figure 11. Schematic diagram of a section of the local winner-take-all circuit. 
Each neuron i receives a unidirectional current input It; the output voltages lri 
represent the result of the local winner-take-all computation. 
To understand the operation of the local winner-take-all circuit, we consider the 
circuit response to a spatial impulse, defined as I : I, where I = lt. It : It-1 
and I : I+1, so v k is much larger than vok_ and vo+, and the resistor circuits 
connecting neuron k with neuron k- 1 and neuron  + 1 saturate. Each resistor 
sinks Io current when saturated; transistor T2 thus conducts 2Io + L current. In 
the subthreshold region of operation, the equation It = Ioexp(V/Vo) describes 
transistor T, and the equation 2Io + L = Io exp((V - vk)/Vo) describes transistor 
710 Lazzaro, Ryckebusch, Mahowald and Mead 
T2k- Solving for vk yields 
vk = Voln((2. + tc)/o) + Vo 
(4) 
As in the original winner-take-all circuit, the output of a winning neuron encodes 
the logarithm of that neuron's associated input. 
As mentioned, the resistor circuit connecting neuron k with neuron k - 1 sinks Io 
current. The current sources lc associated with neurons - 1, - 2, ... must supply 
this current. If the current source lc for neuron  - 1 supplies part of this current, 
the transistor T2k_a carries no current, and the neuron output v'a_ approaches zero. 
In this way, a winning neuron inhibits its neighboring neurons. 
This inhibitory action does not extend throughout the network. Neuron  needs 
only I, current from neurons k- 1, /- 2, .... Thus, neurons sufficiently distant 
from neuron  maintain the service of their current source I,,, and the outputs of 
these distant neurons can be active. Since, for a spatial impulse, all neurons - 1, 
/ - 2, ... have an equal input current I, all distant neurons have the equal output 
v,,, = voln(zc/Zo) + Vo In(Z/So). 
(5) 
Similar reasoning applies for neurons k + 1,/ + 2, .... 
The relative values of I, and L determine the spatial extent of the inhibitory action. 
Figure 12 shows the spatial impulse response of the local winner-take-all circuit, for 
different settings of I/I. 
8 10 12 14 16 
(Position) 
Figure 12. Experimental data showing the spatial impulse response of the local 
winner-take-all circuit, for values of I,/Io ranging over a factor of 12.7. Wider 
inhibitory responses correspond to larger ratios. For clarity, the plots are vertically 
displaced in 0.25 volt increments. 
Winner-Take-All Networks of O(N) Complexity 711 
CONCLUSIONS 
The circuits described in this paper use the full analog nature of MOS devices to 
realize an interesting class of neural computations efficiently. The circuits exploit 
the physics of the medium in many ways. The winner-take-all circuit uses a single 
wire to compute and communicate inhibition for the entire circuit. Transistor Tk 
in the winner-take-all circuit uses two physical phenomena in its computation: its 
exponential current function encodes the logarithm of the input, and the finite 
conductance of the transistor defines the losing output response. As evolution 
exploits all the physical properties of neural devices to optimize system performance, 
designers of synthetic neural systems should strive to harness the full potential of 
the physics of their media. 
Acknowledgments 
John Platt, John Wyatt, David Feinstein, Mark Bell, and Dave Gillespie provided 
mathematical insights in the analysis of the circuit. Lyn Dupr proofread the docu- 
ment. We thank Hewlett-Packard for computing support, and DARPA and MOSIS 
for chip fabrication. This work was sponsored by the Office of Naval Research and 
the System Development Foundation. 
References 
Lazzaro, J.P., Ryckebusch, S., Mahowald, M.A., and Mead, C.A. (1989). Winner- 
Take-All Networks of O(N) Complezity, Caltech Computer Science Department 
Technical Report Caltech-CS-TR-21-88. 
Lazzaro, J.P., and Mead, C.A. (in press). Silicon Models of Auditory Localization, 
Neural Computation. 
Mahowald, M.A., and Delbruck, T.I. (1988). An Analog VLSI Implementation of 
the Marr-Poggio Stereo Correspondence Algorithm, Abstracts of the First Annual 
INNS Meeting, Boston, 1988, Vol. 1, Supplement 1, p. 392. 
Mead, C. A. (in press). Analog VLSI and Neural Systems. Reading, MA: Addison- 
Wesley. 
