678 
A LOW-POWER CMOS CIRCUIT WHICH EMULATES 
TEMPORAL ELECTRICAL PROPERTIES OF NEURONS 
Jack L. Mcador and Clint S. Colc 
Electrical and Computer Engineering Dept. 
Washington Statc Univcrsity 
Pullman WA. 99164-2752 
ABSTRACT 
This paper describes a CMOS artificial neuron. The circuit is 
directly derived from the voltage-gated channel model of neural 
membrane, has low power dissipation, and small layout geometry. 
The principal motivations behind this work include a desire for high 
performance, more accurate neuron emulation, and the need for 
.higher density in practical neural network implementations. 
INTRODUCTION 
Popular neuron models are based upon some statistical measure of known natural 
behavior. Whether that measure is expressed in terms of average firing rate or a 
firing probability, the instantaneous neuron activation is only represented in an 
abstract sense. Artificial electronic neurons derived from these models represent this 
excitation level as a binary code or a continuous voltage at the output of a summing 
amplifier. While such models have been shown to perform well for many applica- 
tions, and form an integral part of much current work, they only partially emulate the 
manner in which natural neural networks operate. They ignore, for example, 
differences in relative arrival times of neighboring action potentials -- an important 
characteristic known to exist in natural auditory and visual networks {Sejnowski, 
1986}. They are also less adaptable to fine-grained, neuron-centered learning, like 
the post-tetanlc facilitation observed in natural neurons. We are investigating the 
implementation and application of neuron circuits which better approximate natural 
neuron function. 
BACKGROUND 
The major temporal artifacts associated with natural neuron function include the 
spacio-temporal integration of synapfic activity, the generation of an action potential 
(AP), and the post-AP hyperpolarization (refractory) period (Figure 1). Integration, 
manifested as a gradual membrane depolarization, occurs when the neuron accumu- 
lates sodium ions which migrate through pores in its cellular membrane. The rate of 
ion migration is related to the level of presynaptic AP bombardment, and is also 
known to be a non-linear function of transmembrane potential. Efferent AP genera- 
tion occurs when the voltage-sensitive membrane of the axosomal hillock reaches 
some threshold potential whereupon a rapid increase in sodium permeability leads to 
A Low-Power CMOS Circuit Which Emulates Neurons 679 
complete depolarization. Immediately thereafter, sodium pores "dose" simultaneously 
with increased potassium permeability, thereby repolarizing the membrane toward its 
resting potential. The high potassium permeability during AP generation leads to the 
transient post-AP hyperpolarization state known as the refractory period. 
v 
Activation 
Threshold 
Figure 1. Temporal artifacts associated with neuron function. (1) gradual 
depolarization, (2) AP generation, (3) refractory period. 
Several analytic and electronic neural models have been proposed which embody 
these characteristics at varying levels of detail. These neuromimes have been used to 
good advantage in studying neuron behavior. However, with the advent of artificial 
neural networks (ANN) for computing, emphasis has switched from modeling neu- 
rons for physiologic studies to developing practical neural network implementations. 
As the desire for high performance ANNs grows, models amenable to hardware 
implementation become more attractive. 
The general idea behind electronic neuromimes is not new. Beginning in 1937 with 
work by Harmon {Harmon, 1937}, lectronic circuits have been used to model and 
study neuronal behavior. In the late 1960's, Lewis {Lewis, 1968} developed a circuit 
which simulated the Hodgkin-Huxley model for a single neuron, followed by 
MacGregor's circuit {MacGregor, 1973} in the early 1970's which modelled a group 
of 50 neurons. With the availability of VLSI in the 1980's, electronic neural imple- 
mentations have largely moved to the realm of integrated circuits. Two different stra- 
tegies have been documented: analog implementations employing operational 
amplifiers {Graf, et al, 1987,1988; Sivilotti, et al, 1986; Raftel, 1988; Schwartz, et al, 
1988}; and digital implementations such as systolic arrays {Kung, 1988}. 
More recently, impulse neural implementations are receiving increased attention. 
Like other models, these neuromimes generate outputs based on some non-linear 
function of the weighted net inputs. However, interneuron communication is realized 
through impulse streams rather than continuous voltages or binary numbers {Murray, 
1988; N. E1-Leithy, 1987}. Impulse networks communicate neuron activation as vari- 
able pulse repetition rates. The impulse neuron circuits which shall be discussed offer 
both small geometry and low power dissipation as well as a closer approximation to 
natural neuron function. 
680 Meador and Cole 
A CMOS IMPULSE NEURON 
An impulse neuron circuit developed for use in CMOS neural networks is shown in 
Figure 2. In thi.q circuit, membrane ion current is modeled by charge flowing to and 
from Ca. Potassium and sodium inflax is represented by current flow from V to the 
capacitor, and ion effiux by flow from the capacitor to ground. The Field Effect- 
Transistors (FETs) connected between V, Vn, and the capacitor emulate voltage- 
and chemically-gated ion channels found in natural neural membrane. In the Figure, 
FET 1 corresponds to the post-synaptic chemically-gated ion channels associated with 
one synapse. FETs 2, 3, and 4 emulate the voltage-gated channels distributed 
throughout a neuron membrane. The following equations summarize circuit opera- 
tion: 
O) 
() 
() 
(4) 
g(t)--h (t)0- (t 4)) 
(5) 
0 if o(t)>; 
Va<Va(t)<Vts and h (Va(t 
h(t)= if vo(t)<o; 
Va<Va(t)<Vts and h(a(t-,))=l 
Vdd 
Excitatory 
Synapse 
I 
L 
1 Io 
Oh(t) 
+ 
va ca I : 
Figure 2. A CMOS impulse neuron with one cxcitatory synapsc-FET. 
Axon 
A Low-Power CMOS Circuit Which Emulates Neurons 681 
Equation (1) expresses how changes in F'a (which emulates instantancons neuron 
excitation) depend upon the sum of three current components controlled by these 
FETs. E, F, and G in equations (2) through (4) express FET drain-source currents as 
functions terminal voltages. Equations (3) and (:5) rely upon the assumption that FET 
2 and FET 3 arc implcmcntcd as a single dual-gate dcvice where the tra_n.conduc- 
tance/2 =/2/3/(/2 +/3). Non-saturatcd FET opcration is assumcd for thcsc cqua- 
tions cvcn though the FETs will momentarily pass through saturation at the onset of 
conduction in the actual circuit. 
The Schmitt trigger circuit establishes a nonlinear positive feedback path responsible 
for action potcntial initiation. Thc upper thrcshold of thc triggcr (V) cmulatcs thc 
natural ncuron activation thrcshold whilc thc lowcr thrcshold (Va) cmulatcs thc max- 
imum hypcrpolarization voltagc. Equation (6) cxprcsscs thc hystcrisis prcscnt in thc 
Schmitt triggcr transfcr charactcristic. Whcn Vs rcachcs thc uppcr Schmitt thrcshold, 
FET 3 turns on, crcating a currcnt path from V to Cs, and cmulating thc upswing of 
a natural action potcntial spikc. A momcnt latcr, FET 2 turns off, starting thc action 
potcntial downswing. Simultancously, FET 4 turns on, begining thc absolutc rcfrac- 
tory period whcrc Cs is dischargcd toward thc maximum hyperpolarization potcntial. 
Whcn that potcntial is rcachcd, thc Schmitt triggcr turns off FET 4 and thc impulse 
firing cyclc is complctc. 
Thc capacitor tcrminal voltagc Va cmulatcs all gross tcmporal artifacts associatcd 
with mcmbranc potcntial, including spacio-tcmporal intcgration, thc action potcntial 
spikc, and a rcfractory period. Thc instantancons nct cxcitation to thc ncuron is 
rcprcscntcd by thc total currcnt flowing into thc summing nodc on thc floating platc 
of thc capacitor. Chargc packcts arc tr_snsfcrrcd from V to the capacitor by thc 
cxcitatory synapse FET. Excitatory packct magnitudc is dcpendcnt upon thc tran- 
sconductance l. Inhibitory synapses (not shown) operatc similarly, but instcad 
rcduce capacitor voltagc by drawing chargc to Vss. A buffcrcd action potential signal 
useful for driving many synapse FETs is availablc at thc gton output. 
Thc mcmbranc potcntial componcnts (E,F, and G) of thc circuit cquations dcscribc 
nonllncar rclationships betwccn post-synaptic cxcitation (E), mcmbranc potcntial (F 
and G), and mcmbranc ion currcnts. Thc functional forms of thcsc componcnts arc 
cquivalcnt to those found bctwccn tcrminal voltagcs and currcnts in non-saturatcd 
FETs. It is notablc that natural voltagc-gatcd channcls do not nccessarily follow thc 
same currcnt-voltagc rclationship of a FET. Evcn though morc accuratc modcls and 
cmulations of natural mcmbranc conductance cxist, it sccms unlikcly at this timc that 
thcy would hclp furthcr improvc ncural network implcmcntation. Thcrc is littic doubt 
that morc complcx circuitry would be rcquircd to bettcr approxlmatc the truc non- 
lincar rclationship found in thc biochemistry of natural ncural mcmbrane. That nccd 
conflicts dircctly with thc goal of high-dcnsity intcgration. 
IMPUIE NEURAL NETWORKS  
Organizing a collection of neuron circuits into a useful network configuration requires 
some weight specification method. Weight values can be either directly specified by 
the designer or learned by the network. A method parlicularly suited for use with the 
fixed FET-synapses of the foregoing circuit is to first learn weights using an off-line  
682 Meador and Cole 
simulation, then translate the numerical results to physical FET transconductances. 
To do this, the activation function of an impulse neuron is derived and used in a 
modified back-propagation learning procedure. 
IMPULSE NEURON ACTIVATION FUNCTION 
Learning algorithms typically require some expression of the neuron activation func- 
tion. Neuron activation can be expressed as a numerical value, a binary pattern, or a 
circuit voltage. In an impulse neuron, activation is expressed in terms of firing rate. 
The more frequently an impulse neuron circuit fires, the greater its activation. 
Impulse neuron activation is a nonlinear function of the excitation imparted through 
its synapse connections. An analytical expression of this nonlinear function can be 
derived using a rectangular approximation of neuron impulse waveforms. 
It is first necessary to define a unit-impulse as one impulse conducted by a synapse 
FET having some pre-determined reference transconductance re/). In Figure 3, To 
represents an invariant activation impulse width which is assumed to be identical for 
all neurons. Tx represents the variable time period required for the neuron to accu- 
mulate the equivalent of K unit-impulses input excitation prior to firing. It can be 
assumed that net input comes from a single excitatory synapse with no other excita- 
tion. It shall also be assumed that impulses arrive at a constant rate, so 
r,=K/wijRi (7) 
where Ri is the firing rate of the source neuron and wij is the weight of the synapse 
connecting neuron i and neuron./. 
The firing rate of the receiving neuron will be Rj= 1/(T0 + T). Substituting for T 
this becomes: 
Rj= 1/(To +K /wijRi)=wqRi/(wijRiTo +K) 
Figure 3 compares this function with the logistic activation function. The impulse 
activation function approaches zero at the rate of 1/K when Ri approaches zero. The 
function also approaches an asymptote of Rj = I/To as Ri increases without bound. 
Any non-synaptic source which causes current flow from l/' to Ca will shift the curve 
to the left, and reflect a spontaneous firing rate at zero input excitation. A similar 
current source to l/' will shift the function to the right, reflecting a positive firing- 
onset threshold. Circuit-level simulations show a dear correspondence to these 
analytical results. This functional form is also evident in activation curves experimen- 
tally observed with natural neurons {Guyton, 1986}. Various natural neurons are 
known to exhibit both spontaneous firing and firing-onset thresholds as well. 
The impulse activation function constant, K, is determined by several factors including 
fire[, Ca, and To. Assuming that T0<<T, no leakage current exists, and that a FET 
conducting in its non-saturated region can be approximated by a resistor, the follow- 
ing expression for K is obtained: 
g = Coin( Wau ) / (mu ) ) / ro (o) 
A Low-Power CMOS Circuit Which Emulates Neurons 683 
where 
Ca is the summing capacitance, V# Vtn are the low and high threshold voltages of the 
Schmitt trigger, and Vn is the gate threshold voltage for an excitatory FET-synapse. 
A more accurate K value can be obtained by using the non-saturated FET current 
equation and solving a nonlinear differential equation. 
1.0 
0.5 
r 
I T1 J Rectangular Impulse Train 
I '1 ' 
Logistic 
Activation / / Alt?vUin 
0 
Figure 3. Rectangular impulse train approximation for impulse activation 
function derivation. Unlike the logistic function which asymptotically 
approaches zero, impulse activation is equal to zero over a range of net 
excitation. 
BACK-PROPAGATION IN IMPUIE NETWORKS 
A back-propagation algorithm has been used to learn connection weights for impulse 
neural networks. At this lime, weight values are non-adaptive (they are fixed at cir- 
cuit fabrication) because they are implemented as invariant FET transconductances. 
Adaptive synapses compatible with impulse neuron circuits are in the early stages of 
development, but are not available at this time. Much can be learned about these net- 
works using non-adaptive prototypes, however. As a result, weight learning is per- 
formed offline as part of the network design process. The back-propagation pro- 
cedure used to learn weights for impulse networks differs from the generalized delta 
rule {Rumelhart, 1986} in two ways. 
The first difference is the use of the impulse activation function instead of the logistic 
function. Any activation nonllnearity is a viable candidate for use with the generalized 
delta rule as long as it is differentiable. This is where difficulties mount with the 
impulse activation function. First of a!l it is not differentiable at zero. What seems to 
be more important, however, is that its first derivative equals zero over a range of 
684 Meador and Cole 
inputs. Examination of the gcncralizcd delta rulc (which performs gradient-descent) 
rcvcals that whcn thc first dcrivativc of ncuron activation becomcs zero, conncctions 
associatcd with that-ncuron will ccasc to adapt. Oncc this happens, thc procedurc will 
most probably ncvcr arrivc at a problcm solution. 
To work around this problcm, a sccond dcviation from the gcncralizcd delta ndc was 
implcmcntcd. This involvcs a dcparturc from using thc truc first dcrivativc whcn thc 
impulse activation becomcs zro. A small constant can be uscd to guarantcc that 
lcarnlng continucs cvcn though thc associatcd ncuron activation is zro: 
Act = 1/(T0 + K /Net) 
0o) 
+I,Z/iVet)' if Net>O (11) 
otherwise 
The use of these equations yields a back-propagation algorithm for impulse networks 
which does not perform true gradient descent, yet which so far has been observed to 
learn solutions to logic problems such as XOR and the 4-2-4 encoder. Investigation 
of other offline learning algorithms for impulse networks continues. Currently, this 
algorithm fulfills the immediate need for an off line procedure which can be used in 
the design of multi-layer impulse neural networks. 
IMPLEMENTATION 
Two requirements for high density integration are low power dissipation and small 
circuit geometry. CMOS impulse neurons use switching circuits having no continuous 
power dissipation. A conventional op-amp circuit must draw constant current to 
achieve linear bias. An op-amp also requires larger circuit geometries for gain accu- 
racy over typical fabrication process variations. Such is not the case for nonlinear 
switching circuits. As a result, these neurons and others like them are expected to 
help improve analog neural network integration density. 
An impulse neuron circuit has been designed which eliminates FETs 2 and 3 of Figure 
2 in exchange for reduced layout area. In this circuit, l/', no longer exhibits an activa- 
tion potential spike. This spike seems irrelevant given the buffered impulse available 
at the axon output. The modified neuron circuit occupies 200 X 25 lambda chip area. 
A fixed FET-synapse occupies a 16 by 18 lambda rectangle. With these dimensions a 
full-interconnect layout containing 40 neurons and 1600 fixed connections will fit on a 
MOSIS 2-micron tiny chip. XOR and 4-2-4 networks of these circuits are being 
developed for 2-micron CMOS. 
CONCLUSION 
The motivation of this work is to improve neural network implementation technology 
by designing CMOS circuits derived from the temporal characteristics of natural neu- 
rons. The results obtained thus far include: 
Two CMOS circuits which dosely correspond to the voltage-gated-charmel 
model of natural neural membrane. 
A Low-Power CMOS Circuit Which Emulates Neurons 685 
Simulations which show that these impulse neurons emulate gross artifacts of 
natural neuron function. 
Initial work on a back-propagation algorithm which learns logic solutions using 
the impulse neuron activation function. 
The development of prototype impulse network I.Cs. 
Future goals involve extending this investigation to plastic synapse and neuron cir- 
cuits, alternate algorithms for both offline and online learning, and practical imple- 
mentations. 
References 
H. P. Graf W. Hubbard L. D. Jackel P. G. N. deVegvar. A CMOS associative 
Memory Chip. IEEE ICNN Con. Proc., pp. 461-468, (1987). 
H. P. Graf L. D. Jackel W. E. Hubbard. VLSI Implementation of a Neural Network 
Model. IEEE Computer, pp. 41-49, (1988). 
A.C. Guyton. Chapt. 10. Organi7ation of the Nervous System: Basic Functions of 
Synapses. Textbook of Physiology, p.136. (1986) 
N. EI-Liethy, R.W. Newcomb, M. Zaghlou. A Basic MOS Neural-Type Junction A 
Perspective on Neural-Type Microsystems. IEEE ICNN Con. Proc., pp. 469-477, 
(1987). 
E. R. Lewis. Using Electronic Circuits to Model Simple Neuroelectric Interactions. 
Proc. IEEE 56, pp. 931-949, (1968). 
R. J. MacGregor R. M. Oliver. A General-Purpose Electronic Model for Arbitrary 
Configurations of Neurons. J. Theor. Biol. 38, pp. 527-538 (1973). 
S. Y. Kung. and J. N. Hwang. Parallel Achitectures for Artificial Neural Nets. IEEE 
ICNN Con. Proc., pp. 11-165 to 11-172, (1988). 
J. Mann R. Lippman B. Berger J. Raftel. A Self-Organizing Neural Net Chip. IEEE 
Cust. Integr. Ckts. Conf., pp. 10.3.1-10.3.5 (1988). 
A. F. Murray A. V. W. Smith. Asynchronous VLSI Neural Networks Using Pulse- 
Stream Arithmetic. IEEE Jnl. of Sol. St. Phys. 23, pp. 688-697, (1988). 
J. I. Raftel. Electronic Implementation of Neuromorphic Systems. IEEE Cust. Integr. 
Ckts. Conf., pp. 10.1.1-10.1.7, (1988). 
D. Rumelhart, G.E. Hinton, and R.J. Williams. Learning Internal Representations by 
Error Propagation. Parallel Distn'buted Processing Vol 1: Foundations, pp. 318-364, 
(1986). 
O. H. Schmitt. Mechanical Solution of the Equations of Nerve Impulse Propagation. 
Am. J. Physiol. 119, pp. 399-400, (1937). 
D. B. Schwartz R. E. Howard. A Programmable Analog Neural Network Chip. IEEE 
Cust. Integr. Ckts. Conf., pp. 10.2.1-1.2.4, (1988). 
686 Meador and Cole 
TJ. Sejnowski. Open Questions About Computation in Cerebral Cortex. Parallel 
Distn'buted Processing Vol. 2:Psychological and Biological Models, pp. 378-385, (1986). 
M. A. Sivilotti M. R. Emerling C. A. Mead. VLSI Architectures for Implementation 
of Neural Networks. Am. Ins. of Phys., 408-413, (1986). 
