A New Constructive Algorithm for Architectural and Functional Adaptation of Artificial Neural Networks

Research output: Contribution to journalArticle

Standard

A New Constructive Algorithm for Architectural and Functional Adaptation of Artificial Neural Networks. / Islam, MM; Sattar, MA; Amin, MF; Yao, Xin; Murase, K.

In: IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Vol. 39, No. 6, 01.12.2009, p. 1590-1605.

Research output: Contribution to journalArticle

Harvard

APA

Vancouver

Author

Bibtex

@article{342cdc8e94794761b2eea558d6fcd30b,
title = "A New Constructive Algorithm for Architectural and Functional Adaptation of Artificial Neural Networks",
abstract = "The generalization ability of artificial neural networks (ANNs) is greatly dependent on their architectures. Constructive algorithms provide an attractive automatic way of determining a near-optimal ANN architecture for a given problem. Several such algorithms have been proposed in the literature and shown their effectiveness. This paper presents a new constructive algorithm (NCA) in automatically determining ANN architectures. Unlike most previous studies on determining ANN architectures, NCA puts emphasis on architectural adaptation and functional adaptation in its architecture determination process. It uses a constructive approach to determine the number of hidden layers in an ANN and of neurons in each hidden layer. To achieve functional adaptation, NCA trains hidden neurons in the ANN by using different training sets that were created by employing a similar concept used in the boosting algorithm. The purpose of using different training sets is to encourage hidden neurons to learn different parts or aspects of the training data so that the ANN can learn the whole training data in a better way. In this paper, the convergence and computational issues of NCA are analytically studied. The computational complexity of NCA is found to be O(W x P-t x t), where W is the number of weights in the ANN, Pt is the number of training examples, and t is the number of training epochs. This complexity has the same order as what the backpropagation learning algorithm requires for training a fixed ANN architecture. A set of eight classification and two approximation benchmark problems was used to evaluate the performance of NCA. The experimental results show that NCA can produce ANN architectures with fewer hidden neurons and better generalization ability compared to existing constructive and nonconstructive algorithms.",
keywords = "constructive approach, generalization ability, functional adaptation, artificial neural networks (ANNs), Architectural adaptation",
author = "MM Islam and MA Sattar and MF Amin and Xin Yao and K Murase",
year = "2009",
month = dec,
day = "1",
doi = "10.1109/TSMCB.2009.2021849",
language = "English",
volume = "39",
pages = "1590--1605",
journal = "IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)",
issn = "1083-4419",
publisher = "Institute of Electrical and Electronics Engineers (IEEE)",
number = "6",

}

RIS

TY - JOUR

T1 - A New Constructive Algorithm for Architectural and Functional Adaptation of Artificial Neural Networks

AU - Islam, MM

AU - Sattar, MA

AU - Amin, MF

AU - Yao, Xin

AU - Murase, K

PY - 2009/12/1

Y1 - 2009/12/1

N2 - The generalization ability of artificial neural networks (ANNs) is greatly dependent on their architectures. Constructive algorithms provide an attractive automatic way of determining a near-optimal ANN architecture for a given problem. Several such algorithms have been proposed in the literature and shown their effectiveness. This paper presents a new constructive algorithm (NCA) in automatically determining ANN architectures. Unlike most previous studies on determining ANN architectures, NCA puts emphasis on architectural adaptation and functional adaptation in its architecture determination process. It uses a constructive approach to determine the number of hidden layers in an ANN and of neurons in each hidden layer. To achieve functional adaptation, NCA trains hidden neurons in the ANN by using different training sets that were created by employing a similar concept used in the boosting algorithm. The purpose of using different training sets is to encourage hidden neurons to learn different parts or aspects of the training data so that the ANN can learn the whole training data in a better way. In this paper, the convergence and computational issues of NCA are analytically studied. The computational complexity of NCA is found to be O(W x P-t x t), where W is the number of weights in the ANN, Pt is the number of training examples, and t is the number of training epochs. This complexity has the same order as what the backpropagation learning algorithm requires for training a fixed ANN architecture. A set of eight classification and two approximation benchmark problems was used to evaluate the performance of NCA. The experimental results show that NCA can produce ANN architectures with fewer hidden neurons and better generalization ability compared to existing constructive and nonconstructive algorithms.

AB - The generalization ability of artificial neural networks (ANNs) is greatly dependent on their architectures. Constructive algorithms provide an attractive automatic way of determining a near-optimal ANN architecture for a given problem. Several such algorithms have been proposed in the literature and shown their effectiveness. This paper presents a new constructive algorithm (NCA) in automatically determining ANN architectures. Unlike most previous studies on determining ANN architectures, NCA puts emphasis on architectural adaptation and functional adaptation in its architecture determination process. It uses a constructive approach to determine the number of hidden layers in an ANN and of neurons in each hidden layer. To achieve functional adaptation, NCA trains hidden neurons in the ANN by using different training sets that were created by employing a similar concept used in the boosting algorithm. The purpose of using different training sets is to encourage hidden neurons to learn different parts or aspects of the training data so that the ANN can learn the whole training data in a better way. In this paper, the convergence and computational issues of NCA are analytically studied. The computational complexity of NCA is found to be O(W x P-t x t), where W is the number of weights in the ANN, Pt is the number of training examples, and t is the number of training epochs. This complexity has the same order as what the backpropagation learning algorithm requires for training a fixed ANN architecture. A set of eight classification and two approximation benchmark problems was used to evaluate the performance of NCA. The experimental results show that NCA can produce ANN architectures with fewer hidden neurons and better generalization ability compared to existing constructive and nonconstructive algorithms.

KW - constructive approach

KW - generalization ability

KW - functional adaptation

KW - artificial neural networks (ANNs)

KW - Architectural adaptation

U2 - 10.1109/TSMCB.2009.2021849

DO - 10.1109/TSMCB.2009.2021849

M3 - Article

C2 - 19502131

VL - 39

SP - 1590

EP - 1605

JO - IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)

JF - IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)

SN - 1083-4419

IS - 6

ER -