Neuronale Architekturen
The TACOMA (TAsk decomposition, COrrelation Measures and local
Attention neurons) Algorithm for Growing Neural Networks
To reduce the engineering efforts for the design of neural network architectures
a data driven algorithm is desirable which constructs a network during the
training process automatically. For structure adaptation different approaches
with evolutionary algorithms, growth algorithms, pruning and others are used. To
solve complex classification or function approximation problems successfully it
is often necessary to divide the problem into subproblems and to solve them
separately. This is a fundamental principle of nature called Task Decomposition.
The resulting key questions are: How to divide a problem automatically in
structured sub-problems and how to build a neural network consisting of
sub-networks reflecting this structure? The answer is quite difficuilt because
what "a good structuring of the problem" is depends on the set of possible
network structures. The idea TACOMA is based on is to build a feed-forward
neural network bottom-up by cyclically inserting cascaded hidden layers. The
activation function of a hidden layer unit combines the local characteristics of
radial basis function units (or other window functions) with sigmoid units:
 |
* |
 |
= |
 |
With each grow step a hidden layer consisting of such local attention neurons
will be inserted. The number of units to be inserted and their attention regions,
means and ratios are given by an approximation of the mapping of the residual
error from output to input space. The attention region of a unit becomes
restricted to a region in the input space where the residual error is still
high. The attention regions of the hidden units of the same hidden layer do not
overlap (no lateral influence) and can be considered as units of different
subnetworks. Contrary to the Cascade-Correlation Learning Architecture different
correlation measures are used to train the units. The hidden units are trained
to maximize the correlation between the units output and the residual error at
the network output. At the same time the hidden units are trained to minimize
the correlation to the other hidden units of the layer. After training a new
hidden layer will be connected to the network using a connection routing
algorithm which connects only cooperative units of different layers. Units of
different hidden layers are called cooperative if their attention regions in the
input space overlap. The TACOMA algorithm genereates over a number of cycles a
neural network well adapted to problem to be solved. The training stops if the
overall error at the network output is small enough. If we now consider the
distribution of the local hidden units in the input space we will find clusters
of cooperative units. Those groups of units can be considered as subnetworks,
each specialized to a different input region. The number of units per subnetwork
expresses the complexity of the subtask in the relating input region and can be
quite different depending on the machine-learning problems.
Benchmarks
Two Spirals Classification Benchmark: The task is to learn to discriminate between two sets of training points which lie on two distinct spirals in the x-y plane. These spirals coil three times around the origin and around one another. This appears to be a very difficult task for back-propagation networks and their relatives. Problems like this one, whose inputs are points on the 2-D plane, are interesting because we can display the 2-D "receptive field" of any unit in the network. The problem was first published by Scott E. Fahlman, CMU.
 |
 |
|
data set |
TACOMA solution |
Two Twin Spirals Classification Benchmark: The task is to learn to discriminate between two sets of training points which lie on two twin spirals in the x-y plane. This appears to be a very difficult (I've never seen a solution other than by TACOMA) task for back-propagation networks and their relatives. Also the solution by the Cascade Correlation Architecture algorithm gives no well generalization. Problems like this one, whose inputs are points on the 2-D plane, are interesting because we can display the 2-D "receptive field" of any unit in the network. The problem was first published by Jan Matti Lange, GFaI.
 |
 |
 |
|
data set |
CASCOR solution |
TACOMA solution |
Publications
- J. M. Lange, H.-M. Voigt, D. Wolf. Task Decomposition and Correlations in Growing Artificial Neural Networks. In M. Mariano, P. G. Morasso, ed. Proceedings of the International Conference on Artificial Neural Networks 94', Sorrento, pages 735-738,Springer, 1994
- J. M. Lange, H.-M. Voigt, D. Wolf. Growing Artificial Neural Networks Based on Correlation Measures, Task Decomposition and Local Attention Neurons. In Proceedings of the IEEE International Conference on Neural Networks 1994 as Part of the IEEE World Congress on Computational Intelligence, Orlando, pages 1355-1358, vol 2, IEEE Press, 1994
- J. M. Lange, H.-M. Voigt, D. Wolf. The TACOMA learning architecture for reflective growing of neural networks In IEE Colloquium on Advances in Neural Networks for Control and Systems, Daimler-Benz AG, Berlin 1994, IEE Digest 94/136, pages 2/1-2/2, The Institution of Electrical Engineers, London, 1994
- H. Paetsch, J. M. Lange. Parallelisierung des TACOMA-Wachstumsalgorithmus fur Neuronale Netze. In Kurzvortrage des 6. Parallelrechner-Anwender-Treffens TAT'94, Institut fur Physiologie der RWTH Aachen, 1994
- J. M. Lange. Cyclical Local Structural Risk Minimization with Growing Neural Networks. Technical Report TR-96-015, International Computer Science Institute, Berkeley, CA, April 1996
Related Work
The TACOMA Architecture is a special neural architecture integrated into the Java Neural Networks Simulator at the University of Tübingen. The implementation for the SNNS system has been done by J. Gatter (Thesis). JavaNNS is freely available.
© GFaI 1994-2005 Kontakt: Dr. H.-M. Voigt
