An improvement of back propagation algorithm using halley third order optimisation method for classification problems

• Main Author: Abdul Hamid, Norhamreeza
• Format: Thesis
• Language: English; English; English
• Published: 2020
• Subjects: QA75-76.95 Calculating machines
• Online Access: http://eprints.uthm.edu.my/923/1/24p%20NORHAMREEZA%20%20ABDUL%20HAMID.pdf; http://eprints.uthm.edu.my/923/2/NORHAMREEZA%20%20ABDUL%20HAMID%20COPYRIGHT%20DECLARATION.pdf; http://eprints.uthm.edu.my/923/3/NORHAMREEZA%20%20ABDUL%20HAMID%20WATERMARK.pdf; http://eprints.uthm.edu.my/923/

Back Propagation (BP) has proven to be a robust algorithm for different connectionist learning problems which commonly available for any functional induction that provides a computationally efficient method. This algorithm utilises first order optimisation method namely Gradient Descent (GD) method which attempts to minimise the error of network. Nevertheless, some major issues need to be considered. The GD method not performed well in large scale applications and when higher learning performances are required. Moreover, it has uncertainty in finding the global minimum of the error function. Besides, they generally depend on the parameters‟ selections. Thus, improving the BP learning efficiency has become an important area of research and consideration specifically in optimisation point of view. The variations of second order optimisation methods have been proposed which provide less iteration of convergence. Yet, an issue with these methods is occasionally converging to the undesired local minima. Inspired by the third order optimisation method which capable to solve unconstrained optimisation problems efficiently in the mathematical research area, this research endeavours to propose a new computational Halley method which is third order optimisation in improving the learning efficiency of BP algorithm namely Halley with Broyden-Fletcher-Goldfarb�Shanno (H-BFGS) and Halley with Davidon-Fletcher-Powell (H-DFP). The efficiency of the proposed methods is compared with the first and second order optimisation method by means of simulation on UCI Machine Learning Repository, Knowledge Extraction Evolutionary Learning and Kaggle dataset. The simulation results show that the highest improvement of H-BFGS in terms of generalisation accuracy is on the Voice Gender classification with 43.33% improvement for 60:40 data division. While H-DFP, the highest improvement achieved in generalisation accuracy is on the Seeds classification with 41.73% improvement for 70:30 data division. Thus, the proposed methods provide significant improvement and promising result in learning Artificial Neural Networks.