Generic Analog Neural Computation 
-- The EPSILON Chip 
Stephen Chumher 
Dept. of Elec. Engineering 
University of Edinburgh 
King's Buildings 
A_ inburgh. EH9 3JL 
Alan F. Murray 
as above 
Donald J. Baxter 
Dept. of Elec. Engineering 
University of Edinburgh 
King's Buildings 
FAinburgh, EH9 3JL 
Alister Hamilton 
Dept. of Elec. Engineering 
University of F_Ainbu_qh 
King's Buildings 
F__inhurgh, EH9 3JL 
H. Martin Reekie 
as above 
Abstract 
An analog CMOS VLSI neural processing chip has been designed and fabri- 
cated. The device employs "pulse-stream" neural state siLmalling, and is capa- 
ble of computing some 360 million symaptic connections per second. In addi- 
tion to basic characterisation results. the performance of the chip in solving 
"real-world" problems is also demonstrated. 
1 INTRODUCTION 
Inspired by biology. and home out of a deshe to perform analogue computation with fun- 
damentally digital fabrication processes, the so-called "pulse-stream" arithmetic system 
has been steadily evolved and improved since its inception in 1986 (Murray1990a, Mur- 
ray1989a). In addition to this continuous development at Edinburgh, many other research 
groups around the world (most notably Meador et al (Meador1990a)) have experimented 
with their own pulse-firing neural circuits. 
In pulsed implementations, each neural state is represented by some variable attribute 
(e.g. the width of fixed fieXlUency pulses, or the rate of fixed width pulses) of a train (or 
"stream") of pulses. The neuron design therefore reduces to a form of oscillator. Each 
neuron is fed by a column of syaapses, which multiply incoming neural States by the 
synaptic weights. In contrast with the original circuits of Murray and Smith (Mur- 
ray1987a), the synapse design which will be discussed herein utilises analog circuit tech- 
niqnes to perform the multiplication of neural state by synal>tic weight. 
This paper describes the Edinburgh Pulse-Stream Implementation of a Learning Oriented 
Network (EPSILON) chip. EPSILON was developed as a flexible neural processor, capa- 
ble of addressing a variety of applications. The main design criteria were as follows: 
 That it be large enough to be of use in practical problems. 
 It should be capable of implementing networks of arbitrary size and architecture. 
 It must be able to act as both a "slave" accelerat to a conventional computer, and as 
an "auto!lomous" procsor. 
As will be seen, these constraints resulted in a chip which could realise only a single lay- 
er of synaptic weights, but which could be cascaded to form large, useful networks for 
solving real-world problems. 
773 
774 Churcher, Baxter, Hamilton, Murray, and Reekie 
The remaining sections of this paper describe the attributes of pulse-coded neural systems 
in general, before detailing the circuits which were employed on EPSIIX)N. Finally. re- 
suits from a vowel recognition application are presented, in order to illustrate the perfor- 
mance of EPSH.DN when applied to real tasks. 
2 PULSE CODED NEURAL SYSTEMS 
As already mentioned, EPSILON is a pulse coded analog neural processing chip. In such 
implementations, neural sUtes are encoded as digital pulses. The sutes themselves may 
then be represented either by varying the width of the pulses (pulse width modulation -- 
PWM), or by varying the rate of the pulses (pulse frequency modulation -- PFM). The 
arguments for using pulses in this way are strong. Firsfly, they provide a very effective 
and robust method for communicating sutes both on- and between-chip, since pulses are 
extremely resistant to noise. Secondly, the use of pulses to represent states renders inter- 
facing to digital circuits and computer peripherals straightforward. Finally, pulsed sig- 
nalling leads to simplification of artlhmetic circuits (i.e. synapses), resulting in much 
higher inter-connection densities. 
Unfortunately, pulse-based systems do have drawbacks. In common with all analog cir- 
cuits, the synaptic computing elements have limited precision (usually equivalent to 
about 7 bits), and their performance is subject to the vagaries of fabrication process varia- 
tions. This results in a situation whereby supposedly "matched" circuits vary markedly in 
theit characteristics. Furthermore, the switching which is inherent in any pulsed circuit 
results in increased levels of system noise, most usna!!y in the form of power supply tran- 
sients. An additional problem with pulse frequency modulation (PFM) systems is that 
computation rates are dependent on the data; this is an important consideration in speed- 
critical applications. 
3 THE EPSILON DESIGN 
This section describes the circuits which were used in the EPSILON design. The operat- 
ing principles of each circuit are discussed, and characterisation results presented. In ac- 
cordance with the demerits mentioned in the previous section. all circuits were designed 
to be tolerant to noise and process variations, to cause as little noise as possible them- 
selves, and to be easy to "set up" in practice. Finally, the specification of the EPSILON 
chip is presented. 
3.1 SYNAPSE 
The synapse design was based on the standard transconductance multiplier circuit, which 
had previously been the basis for monolithic 8nalogue transversal filters for use in signal 
processing applications (Denyer1981a). Such multipliers use MOS transistors in their 
linear region of operation to generate output currents proportional to a product of two in- 
put voltages. This concept was adapted for use in pulsed neural networks by fLxing one of 
the input voltages, and using a neural sure to gate the output current. In this manner, the 
synapfic weight controls the magnitude of the output current, which is multiplied by the 
incoming neural pulses. The resultant charge packets are subsequently integrated to yield 
the total post-synaptic activity voltage. 
Figure 1 shows the basic pulsed multiplier cell, where M1 and M2 form the transconduc- 
tance multiplier, and M3 is the output pulse transistor. By ensuring that the drain-source 
voltages for M1 and M2 are the same and constant (the differential amplifier and transis- 
tors M4 and M5 are used to satisfy this constraint), non-linearifies in the transistor re- 
sponses can be cancelled ont, such that 1ov r is linearly dependent on the difference of 
Vas and Vas,, (Murray1992a). Multiplication is achieved by pulsing thi. currem by the 
neural state, V. An "instantaneous" representation of the aggregated post-synaptic activ- 
ity is given by the output voltage, Voor; this must subsequently be integrated in order to 
provide an activity input to a neuron. 
Generic Analog Neural Computation--The EPSILON Chip 775 
Figure 1: Transconductance Multiplier Synapse 
14.0 
 13.0 
 12.0 
.! 11.0 
_o 10.0 
 9.0 
o 8.0 
7.0 
0.0 90.0 
I I I I I I I 
4.7 
4.4 
4.1 
 '--- 3.75 
3.4 
3.1 
2.8 
I I I  I I I I 
10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 
Input State (microseconds) 
Figure 2: Synapse Characterisation Results 
Results from characterisation tests of the synapse are presented in Figure 2, which shows 
output state against input state, for different synapfic weight voltages. As seen from the 
Figure, the linearity of the synapses, with respect to input state, is very high. The varia- 
tion of synapse response with synaptic weight voltage is also fairly uniform. The graphs 
depict mean performance over all the synaptic columns in all the chips tested. The associ- 
ated standard deviations were more or less constant, representing a variation of approxi- 
mately + 300 ns in the values of the output pulse widths. The effects of intra- and inter- 
chip process mismatches would therefore seem to be well contained by the circuit design. 
The "zero point" in the synaptic weight range was set at 3.75 V and, as can be seen from 
the Figure, each graph shows an offset problem when the input neural state is zero. This 
was attributable to an imbalance in the operating conditions of the transistors in the 
synapse, induced by the non-ideal nature of the power supplies (i.e. the non-zero sheet 
resistance of the power supply tracks), resulting in an offset in the input voltage to the 
post-synaptic integrator. This problem is easily obviated in practice, by employing three 
synapses per column to cancel the offset. 
776 Churcher, Baxter, Hamilton, Murray, and Reekie 
3.2 NEURONS 
In order to reflect the diversity of neural network forms. and possible applications. two 
different neuron desi.q were included on the EPSILON chip. The first, a synchronous 
pulse width modulation neuron was desitmed with vision applications in mind. This cir- 
cuit could guarantee network computation times, thereby eliminating the data dependen- 
cy inherent in pulse frequency systems. The second neuron design used asynchronous 
pulse frequency modulation; the asynchronous nature of these circuits is advantageous in 
feedback and recurrent neural architectures, where temporal characteristics are important. 
As with the synapse, both circuits were designed to miniraise transient noise injection, 
and to be tolerant of process variations. 
3.2.1 Pulse Width Modulation 
As already stated, this system retains all the advantages of using pulses for commuaica- 
fion/calculation, whilst being able to guarantee a maximum network evaluation time. In 
the first instance. the main disadvantage with this technique appeared to be its syn- 
chronous nature - neurons would all be switching together causing larger power supply 
transients than in an asynchronous system. This problem has, however. been circumvent- 
ed via a "double-sided" pulse modulation scheme. which will be more fully explained lat- 
VOT 
VOU- 1' 
Figure 3: Pulse-Width Modulation Neuron 
The operation of the pulse-width modulation neuron is illustrated in Figure 3. The neuron 
itseft is nothing more elaborate than a 2-stage comparator, with an inverter output driver 
stage. The inputs to the circuit are the integrated post-syaaptic activity voltage, VAcr, and 
a reference voltage, Vp, which is generated off-chip and is globally distributed to all 
neurons in parallel. As seen from the waveforms in Figure 3, the output of the neuron 
changes state whenever the reference signal crosses the activity voltage level. An output 
pulse. which is some function of the input activation, is thus generated. The transfer func- 
tion is entirely dependent on the shape of the reference signal- when this is generated by 
a RAM look-up table, the function can become completely arbitrary and hence user pro- 
grammable. Figure 3 shows the signal which should be applied if a sigmoidal transfer 
characteristic is desired. Note that the sigmoid si,nsls are "on their sides" - this is be- 
cause the input (or independent variable) is on the vertical axis rather than the horizontal 
axis, as would normally be expected. The use of a "double-sided" ramp for the reference 
signal was alluded to earlier - this mechanism generates a pulse which is symmetrical 
about the mid-point of the ramp, thereby greatly reducing the likelihood of coincident 
Generic Analog Neural Computation--The EPSILON Chip 777 
edges. This edge-asynchronicity obviates the problem of larger switching transients on 
the power supplies. Furthermore, because the analogue element (i.e. the ramp voltage) is 
effectively removed from the chip, and the circuit itself merely functions as a digital 
block, the system is immune to process variations. 
100 
80 
20 
0 
I. '" 
,/:.? 1.0 -- _ 
.... 
......-.:'. , 0.6 --- 
..-"',-' ." / 
1.6 1.8 2 2.2 2.4 2.6 2.8 3 3.2 3.4 
Activity (V) 
Figure 4: PWM Neuron Performance 
Figure 4 shows plots of output state (measured as a percentage of a maximum possible 
20 #s pulse) versus input activity voltage, for five different sig'moid "temperatures", aver- 
aged over all the neurons on one chip. As can be seen, the fidehty of the sigmoids is ex- 
tremely high, and it should be noted that all the curves are symmetrical about their mid- 
points -- something which is difficult to achieve using standard analog circuits. 
3.2.2 Pulse Frequency Modulation 
The second neuron design which was included in EPSILON used pulse frequency encod- 
ing of the neural state. Although hampered by data dependent calculation times, its whol- 
ly asynchronous nature makes it ideal for neural network architectures which embody 
temporal characteristics i.e. feedback networks, and recurrent networks. 
VIH ID 
 Vdd MIO  Vdd 
4 ID 
] vj 
Figure 5: Pulse Frequency Modulation Neuron 
778 Churcher, Baxter, Hamilton, Murray, and Reekie 
The neuron design is illustrated in Figure 5, and is basically a Voltage Controlled Oscilla- 
tor (VCO) with a variable gain sigmoidal transfer characteristic. Oscillation is achieved 
via the hysteretic charge and discharge of capacitor C. by the currents IH and IL respec- 
tively. The output pulse width is constant, and is set by IH. whilst the inter-pulse spacing 
(and hence output frequency) is controlled by IL. IL itself is determined by the activity 
voltage. VXI, via the differential stage constituted by transistors M3 to M6. It is this latter 
which gives the VCO its sigmoidal characteristic. and gain variations may be achieved by 
injecting and removing additional current at appropriate points in this stage (note that the 
circuitry for this has been omitted from Figure 5. for the sake of clarity). 
1o 
o 
I 
-0.$ 0 0. 
-1 I 1.5 
Figure 6: PFM Neuron Performance 
The characterisation results for the VCO are presented in Figure 6. The Figure shows 
plots of output percentage duty cycle versus input differential voltage. for different values 
of sigmoid gain. Note that the curves are fair approximations to sigmoids, although. in 
contrast with the pulse width modulation neuron, they are not symmetrical about their 
mid-points. It can also be seen that the range of possible sigmoid gains is smaller than the 
range available with the PWM system, although this is not a crucial factor in many appli- 
cations. 
3.3 EPSILON SPECIFICATION 
The circuits described in the previous section were combined to form the EPSILON chip. 
This was subsequently fabricated by European Silicon Smcmres (ES2) using their 
ECPD15 (i.e. 1.5/an, double metal, single poly CMOS). As already stated, each chip was 
capable of implementing a single layer of synaptic connections, and could accept inputs 
as either analog voltages (for direct interface to sensors) or as pulses (for communication 
with other chips, and with digital systems). The full specification is given in Table 1. 
4 APPLICATION -- VOWEL RECOGNITION 
After the device characterisation experiments had been completed, EPSILON was used to 
implement a multi-layer perceptton (MLP) for speech data classification. The MI,P had 
54 inputs, 27 hidden units, and 11 outputs, and the task was to classify 11 different vowel 
sounds spoken by each of 33 speakers. The input vectors were formed by the analog out- 
puts of 54 band-pass filters. 
The MI.P was initially trained on a SPARC station, using a subset of 22 patterns. Learn- 
ing (using the Virtual Targets algorithm, with 0 % noise (Murray1991a)) proceeded until 
the maximum bit error in the output vector was < 0. 3, at which point the weight set was 
Generic Analog Neural Computation--The EPSILON Chip 779 
Table 1: EPSILON Specifications 
EPSILON S .cification 
No. of State Input Pins 
No. of Actual State Inputs 
Input Modes 
No. of State Outputs 
30 
120. Muxed in Banks of 30 
Analog. PW. or PF 
30, Directly Pinned Out 
Output Modes 
No. of Synapses 
No. of Weight Load Channels 
Weight Load Tune 
Weight Storage 
Maximum Speed (cps) 
Technology 
Die Size 
Maximum Power Dissipation 
PW or PF 
3600 
2 
3.6 ms 
Dynamic 
360 Mcps 
1.5 #m, Double Metal CMOS 
9.5 mmx 10. 1 mm 
350 mW 
 0.3 
[- 0.25 
0.2 
0.15 
0.05 
0 
0 200 400 600 800 1000 1200 1400 
Epoch Number 
Figure 7: EPSILON Under Training 
downloaded to EPSILON. Training was then restarred under the same regime as before 
(using the same "stop" criterion), although this time EPSILON was used to evaluate the 
"forward pass" phases of the network. Figure 7 shows the evolution of mean square error 
with number of epochs during this period; at the end of llmining, EPSILON could correct- 
ly identify all 22 training patterns. 
Subsequent to this, 176 "unseen" test patterns were presented to the EPSILON network, 
with the result that 65.34 % of these vectors were correctly classified. This compared 
very favourably with similar generalisation experiments which were carried out on a 
SPARC: in this case, the best result obtained was 67.61%. 
5 CONCLUSIONS 
In conclusion, a large analog VLSI neural chip, composed of process tolerant circuits, 
with useful characteristics has been fabricated. Although not a self-learning device, it has 
been proved that EPSILON will support learning, and can be applied successfully to real- 
780 Churcher, Baxter, Hamilton, Murray, and Reekie 
world problems. Indeed, when correctly trained. the performance of the EPSILON chip 
has been shown to be comparable with that of software simulations on a SPAaRC station. 
Work is currently under way to apply EPSILON to computer vision tasks: more specifi- 
cally, it it will be used to implement an MLP which is capable of recognisiag regions in 
segmented images of natural scenes. Furthermore, the success of the learning experi- 
ments has given us sufficient confidence to undertake the development of a serf-learning 
analog neural chip. It is envisaged that this will employ EPSILON-type circuits, and will 
implement the Virtual Targets (Murray1991a) training algorithm; the design of a small 
prototype is currently nearing completion. 
Acknowledgements 
The authors would like to thank the Science and Engineering Research Council for their 
continued funding of  work. In addition, Stephen Chumher and Donald Baxter are 
grateful to British Aerospace PLC and Thom-EMI CRL respectively for sponsor.hip and 
technical supgxxt during the course of their PhI)'s. 
Lionel Tarassenko (Dept. of Eagineering Science, University of Oxford) must also be 
thanked for his invaluable comments, and for supplying the vowel database. 
References 
Murray 1990a. 
A. E Murray, D. Baxter, Z. Butler, S. Churcher, A. Hmilton, H. M. Reekie, and L. 
Tarassenko. "Innovations in Pulse Stream Neural VLSI: Arjthm,tic and Commu- 
nications", IEEE Workshop on Microelectronics for Neural Networks, Dortmund 
1990, pp. 8-15, 1990. 
Murray 1989a. 
A. E Murray. "Pulse Arithmetic in VLSI Neural Networks", IEEE MICRO, vol. 9, 
no. 6, pp. 64-74, 1989. 
Meador 1990a, 
J. Meador, A. Wu, C. Cole, N. Ninmnze, and P. Chintrakulchai, "Programmable 
Impulse Neural Circuits", IEEE Transactions on Neural Networks, vol. 2, no. 1, 
pp. 101-109. 1990. 
Murray 1987a. 
A. E Murray and A. V. W. Smith. "Asynchronous Arithmetic for VLSI Neural 
Systems", Electronics Letters, vol. 23, no. 12, pp. 642-643, June, 1987. 
Denyet 1981 a. 
P. B. Denyer and J. Mav, "MOST Transconductance Multipliers for Array Appli- 
cations", lEE Proc. Pt. 1, vol. 128, no. 3. pp. 81-86, June 1981. 
Murray 1992a. 
A.E Murray, A. Hamilton, D.J. Baxter. S. Chumher. H.M. Reekie. and L. 
Tarassenko, "Integrated Pulse-Stream Neural Networks - Results, Issues and Point- 
ers", IEEE Trans, Neural Networks, pp. 385-393, 1992. 
Murray 1991a. 
A. E Murray. "Analog VLSI and Multi-Layer Perceptrons - Accuracy, Noise and 
On-Chip Learning". Proc, Second International Conference on Microelectronics 
for Neural Networks, Munich (Germany), pp. 27-34. 1991. 
