A Winner-Take-All Circuit with 
Controllable Soft Max Property 
Shih-Chii Liu 
Institute for Neuroinformatics, ETH/UNIZ 
Winterthurstrasse 190, CH-8057 Zurich 
Switzerland 
shih@ini.phys.ethz.ch 
Abstract 
I describe a silicon network consisting of a group of excitatory neu- 
rons and a global inhibitory neuron. The output of the inhibitory 
neuron is normalized with respect to the input strengths. This out- 
put models the normalization property of the wide-field direction- 
selective cells in the fly visual system. This normalizing property is 
also useful in any system where we wish the output signal to code 
only the strength of the inputs, and not be dependent on the num- 
ber of inputs. The circuitry in each neuron is equivalent to that in 
Lazzaro's winner-take-all (WTA) circuit with one additional tran- 
sistor and a voltage reference. Just as in Lazzaro's circuit, the 
outputs of the excitatory neurons code the neuron with the largest 
input. The difference here is that multiple winners can be chosen. 
By varying the voltage reference of the neuron, the network can 
transition between a soft-max behavior and a hard WTA behav- 
ior. I show results from a fabricated chip of 20 neurons in a 1.2pm 
CMOS technology. 
1 Introduction 
Lazzaro and colleagues (Lazzaro, 1988) were the first to implement a hardware 
model of a winner-take-all (WTA) network. This network consists of N excitatory 
cells that are inhibited by a global signal. Improvements of this network with ad- 
dition of positive feedback and lateral connections have been described (Morris, 
1998; Indiveri, 1998). The dynamics and stability properties of networks of cou- 
pled excitatory and inhibitory neurons have been analyzed by many (Amari, 1982; 
Grossberg, 1988). Grossberg described conditions under which these networks will 
exhibit WTA behavior. Lazzaro's network computes a single winner as reflected by 
the outputs of the excitatory cells. Several winners can be chosen by using more 
localized inhibition. 
In this work, I describe two variants of a similar architecture where the outputs of 
the excitatory neurons code the relative input strengths as in a soft-max compu- 
tation. The relative values of the outputs depend on the number of inputs, their 
relative strengths and two parameter settings in the network. The global inhibitory 
718 S.-C. Liu 
ei_ 1 ei ei+ 1 
Yi-1 
wi. 
Yi+l 
Wi+l 
Figure 1: Network model of recurrent inhibitory network. 
signal can also be used as an output. This output saturates with increasing num- 
ber of active inputs, and the saturation level depends on the input strengths and 
parameter settings. This normalization property is similar to the normalization be- 
havior of the wide-field direction-selective cells in the fly visual system. These cells 
code the temporal frequency of the visual inputs and are largely independent of the 
stimulation size. The circuitry in each neuron in the silicon network is equivalent 
to that in Lazzaro et. al.'s hard WTA network with an additional transistor and 
a voltage reference. By varying the voltage reference, the network can transition 
between a soft-max computation and a hard WTA computation. In the two vari- 
ants, the outputs of the excitatory neurons either code the strength of the inputs 
or are normalized with respect to a constant bias current. Results from a fabri- 
cated network of 20 neurons in a 1.2/zm AMI CMOS show the different regimes of 
operation. 
2 Network with Global Inhibition 
The generic architecture of a recurrent network with excitatory neurons and a single 
inhibitory neuron is shown in Figure 1. The excitatory neurons receive an external 
input, and they synapse onto a global inhibitory neuron. The inhibitory neuron, in 
turn, inhibits the excitatory neurons. The dynamics of the network is described as 
follows: 
N 
dyi 
dt = -Yi + ei - g(E wjyj) (1) 
j=l 
where wj is the weight of the synapse between the jth excitatory neuron and the 
inhibitory neuron, and yj is the state of the jth neuron. Under steady-state condi- 
N 
tions, Yi = ei -- YT, where YT = g(j= wjyj). 
Assume a linear relationship between YT and yj, and letting wj - w, 
N -V= 1 ej 
YT = w E YJ -- w 
l+wN 
j=l 
As N increases, YT -- --1 e 
N . If all inputs have the same level, e, then YT -- e. 
A Yrinner-Take-All Circuit with Controllable Soft Max Property 719 
Iol 
VT 
Vr2 
Figure 2: First variant of the architecture. Here we show the circuit for two excita- 
tory neurons and the global inhibition neuron, M4. The circuit in each excitatory 
neuron consists of an input current source, I, and transistors, M to M3. The 
inhibitory transistor is a fixed current source, Ib. The inputs to the inhibitory 
transistor, Io and I;2 are normalized with respect to Ib. 
3 First Variant of Network with Fixed Current Source 
In Sections 3 and 4, I describe two variants of the architecture shown in Figure 
1. The two variants differ in the way that the inhibition signal is generated. The 
first network in Figure 2 shows the circuitry for two excitatory neurons and the 
inhibition neuron. Each excitatory neuron is a linear threshold unit and consists of 
an input current, I, and transistors, M, M2, and M3. The state of the neuron is 
represented by the current, Ir. The diode-connected transistor, M, introduces a 
rectifying nonlinearity into the system since It1 cannot be negative. The inhibition 
current, IT, is sunk by M, and is determined by the gate voltage, VT. The inhibition 
neuron consists of a current source, Ib, and VT is determined by the corresponding 
current, I and the corresponding transistor, Ms in each neuron. Notice that IT 
cannot be greater than the largest input to the network and the inputs to this 
network can only be excitatory. The input currents into the transistor, M4, are 
defined as Io and lo2 and are normalized with respect to the current source, Ib. In 
the hard WTA condition, the output current of the winning neuron is equal to the 
bias current, I,. 
This network exhibits either a soft-maximum behavior or a hard WTA behavior 
depending on the value of an external bias, Va. The inhibition current, IT, is 
derived as: 
INIi Nil 
- - (2) 
+ + N 
where N is the number of "active" excitatory neurons (that is, neurons whose 
Ii > IT), Ii is the same input current to each neuron, and Is - Ioe v/r. In 
deriving the above equation, we assumed that  - 1. The inhibition current, IT, is 
a linear combination of the states of the neurons because IT = Y. Ii x I/I. 
Figure 3(a) shows the response of the common-node voltage, VT, as a function of 
the number of inputs for different input values measured from a fabricated silicon 
network of 20 neurons. The input current to each neuron is provided by a pFET 
transistor that is driven by the gate voltage, V/n. All input currents are equal in 
this figure. The saturation behavior of the network as a function of the number 
720 S.-C. 
o. O.o% 
Number of inpu Number of inputs 
(a) (b) 
Figure 3: (a) Common-node voltage, VT, as a function of the number of input 
stimuli. Va = 0.8V. (b) Common-node voltage, VT, as a function of the number 
of inputs with an input voltage of 4.3V and Vb - 0.7V. The curves correspond to 
different values of Va. 
of inputs can be seen in the different traces and the saturation level increases as 
n decreases. As seen in Equation 2, the point at which the response saturates is 
dependent on the ratio, It,/Ia. In Figure 3(b), I show how the curve saturates at 
different points for different values of Va and a fixed Ib and V/n. 
In Figure 4, I set all inputs to zero except for two inputs, n and Vin2 that are set 
to the same value. I measured Io and Io as a function of Va as shown in Figure 
4(a). The four curves correspond to four values of n. Initially both currents Io 
and Io2 are equal as is expected in the soft-max condition. As Va increases, the 
network starts exhibiting a WTA behavior. One of the output currents finally goes 
to zero above a critical value of Va. This critical value increases for higher input 
currents because of transistor backgate effects. In Figure 4(b), I show how the 
output currents respond as a function of the differential voltage between the two 
inputs as shown in Figure 4. Here, I fixed one input at 4.3V and swept the second 
input differentially around it. The different curves correspond to different values of 
Va. For a low value of Va, the linear differential input range is about 100mV. This 
linear range decreases as Va is increased (corresponding to the WTA condition). 
4 
Second Variant with Diode-Connected Inhibition 
Transistor 
In the second variant shown in Figure 5, the current source, M4 is replaced by a 
diode-connected transistor and the output currents, Ioi, follow the magnitude of 
the input currents. The inhibition current, IT, can be expressed as follows: 
I T : (IrilIoi) X Ia 
(s) 
where Ia is defined in Section 3. We sum Equation 3 over all neurons and assuming 
equal inputs, we get IT = v/ Iri x Ia. This equation shows that the feedback 
signal has a square root dependence on the neuron states. As we will see, this 
causes the feedback signal to saturate quickly with the number of inputs. 
A I4qnner-Take-All Circuit with Controllable Soft Max Property 721 
Vin=4.3V 4 2V4 1 p 
: ? 4'5/ l 
0.5 0.6 0.7 0.8 
Va (v) 
2.5 
10 -7 
Va---O.7V 
-0.2 
?;' - Va--O.4V 
-0.1 0 0.1 0.2 0.3 
Vin2-Vin 1 (V) 
(a) (b) 
Figure 4: (a) Output currents, Io and Io2, as a function of V*` for a subthreshold 
bias current and n - 4.0V to 4.3V. (b) Outputs, Io and Io2, as a function of the 
differential input voltage, AV/n, with n = 4.3V. 
Irl I 
_- 
() I2 
Ir2 
Figure 5: Second variant of network. The schematic shows two excitatory neurons 
with diode-connected inhibition transistor. 
Substituting Iri = Ii - IT in Equation 3, we solve for IT, 
IT = -I*`N + I 
N 
(I*`N) 2 + 41,, EIi (4) 
i 
From measurements from a fabricated circuit with 20 neurons, I show the depen- 
dence of VT (the natural logarithm of IT) on the number of inputs in Figure 6(a). 
The output saturates quickly with the number of inputs and the level of saturation 
increases with increased input strengths. All the inputs have the same value. 
The network can also act as a WTA by changing V*`. Again, all inputs are set to 
zero except for two inputs whose gate voltages are both set at 4.2V. As shown in 
Figure 6(b), the output currents, Io and Ion, are initially equal, and as V*` increases 
above 0.6V, the output currents split apart and eventually, Io2 - 0A. The final value 
of lol depends on the maximum input current. This data shows that the network 
acts as a WTA circuit when V*` > 0.73V. If I set V/n2 - 4.25V instead, the output 
currents split at a lower value of V*`. 
722 S.-C. Liu 
0.45 
0.4 
0.35 
0.3 
0.25 
Vin=3.9V 
 ..... Vin=4.06V 
';/ Vin--4.3V 
i/ 
// 
0'20  z  i 1'0 12 
Number of inputs 
(a) 
.-2 
x 10 -? 
Vinl=4.2V, Vin2=4.25V 
':  Vin l=Vin2.2V 
Va (V) 
Figure 6: (a) Common-node voltage, VT, as a function of the number of inputs for 
input voltages, 3.9V, 4.06V, and 4.3V for V = 0.4V. (b) Outputs, Ioi and lo2, as 
a function of V for hi = 4.2V, n2 - 4.25V for the 2 curves with asterisks and 
for Vnl - he -- 4.2V for the 2 curves with circles. 
5 Inhibition 
The WTA property arises in both variants of this network if the gain parameter, 
V, is increased so that the diode-connected transistor, M2, can be ignored. Both 
variants then reduce to Lazzaro's network. In the first variant, the feedback current 
(IT) is a linear combination of the neuron states. However, when the gain parameter 
is increased so that Me can be ignored, the feedback current is now a nonlinear 
combination of the input states so the WTA behavior is exhibited by these reduced 
networks. 
Under hard WTA conditions, if IT is initially smaller than all the input currents, the 
capacitances C at the nodes Vrl and Vr2 are charged up by the difference between 
the individual input current and IT, i.e., av. _ i-r. Since the inhibition current 
dt -- C 
is a linear combination of Iri and Iri is exponential in Vr, we can see that I is 
a sum of the exponentials of the input currents, I. Hence the feedback current is 
nonlinear in the input currents. Another way of viewing this condition in electronic 
terms is that in the soft WTA condition, the output node of each neuron is a soft- 
impedance node, or a low-gain node. In the hard WTA case, the output node is now 
a high-impedance node or a high-gain node. Any input differences are immediately 
amplified in the circuit. 
6 Discussion 
Hahnloser (Hahnloser, 1998) recently implemented a silicon network of linear thresh- 
old excitatory neurons that are coupled to a global inhibitory neuron. The inhibitory 
signal is a linear combination of the output states of the excitatory neurons. This 
network does not exhibit WTA behavior unless the excitatory neurons include a 
self-excitatory term. The inhibition current in his network is also generated via a 
diode-connected transistor. The circuitry in two variants described here is more 
compact than the circuitry in his network. 
Recurrent networks with the architecture described in this paper have been proposed 
by Reichardt and colleagues (Reichardt, 1983) in modelling the aggregation property 
A Vqnner-Take-All Circuit with Controllable Soft Max Property 723 
of the wide-field direction-selective cells in flies. The synaptic inputs are inhibited 
by a wide-field cell that pools all the synaptic inputs. Similar networks have also 
been used to model cortical processing, for example, orientation selectivity (Douglas, 
1995). 
The network implemented here can model the aggregation property of the direction- 
selective cells in the fly. By varying a voltage reference, the network implements 
either a soft-max computation or a hard WTA computation. This circuitry will be 
useful in hardware models of cortical processing or motion processing in inverte- 
brates. 
Acknowledgments 
I thank Rodney Douglas for supporting this work, and the MOSIS foundation for 
fabricating this circuit. I also thank Tobias Delbrfick for proofreading this docu- 
ment. This work was supported in part by the Swiss National Foundation Research 
SPP grant and the U.S. Office of Naval Research. 
References 
Amari, S., and Arbib, M. A., "Competition and cooperation in neural networks," 
New York, Springer-Verlag, 1982. 
Grossberg, W., "Nonlinear neural networks: Principles, mechanisms, and architec- 
tures," Neural Networks, 1, 17-61, 1988. 
Hanhloser, R., "About the piecewise analysis of networks of linear threshold neu- 
rons," Neural Networks, 11,691-697, 1988. 
Hahnloser, R., "Computation in recurrent networks of linear threshold neurons: 
Theory, simulation and hardware implementation," Ph.D. Thesis, Swiss Federal 
Institute of Technology, 1998. 
Lazzaro, J., Ryckebusch, S. Mahowald, M.A., and Mead. C., "Winner-take-all net- 
works of 0(n) complexity," In Tourestzky, D. (ed), Advances in Neural Information 
Processing Systems 1, San Mateo, CA: Morgan Kaufman Publishers, pp. 703-711, 
1988. 
Morris, T.G., Horiuchi, T. and Deweerth, S.P., "Object-based selection within an 
analog VLSI visual attention system," IEEE Trans. on Circuits and Systems II, 
45:12, 1564-1572, 1998. 
Indiveri, G., "Winner-take-all networks with lateral excitation," Neuromorphic Sys- 
tems Engineering, Editor, Lande, TS., 367-380, Kluwer Academic, Norwell, MA, 
1998. 
Reichardt, W., Poggio, T., and Hausen, K., "Figure-ground discrimination by rel- 
ative movement in the visual system of the fly," Biol. Cybern., 46, 1-30, 1983. 
Douglas, RJ., Koch, C., Mahowald, M., Martin, KAC., and Suarez, HH., "Recurrent 
excitation in neocortical circuits," Science, 269:5226, 981-985, 1995. 
