Dendronic Decisions Limited

Advanced Computing Research -- Expertise in Machine Learning

Articles

Introduction

Adaptive Logic Networks Technical Overview

Publications

Introduction

Adaptive Logic Networks technology has recently emerged as an effective alternative to artificial neural networks for machine learning tasks. This technical overview describes the advantages of Adaptive Logic Networks technology and its diverse applications of analysis, prediction, and control. To fully motivate the discussion of Adaptive Logic Networks, a brief background on the goals and benefits of machine learning, or neurocomputing, is presented.

Neurocomputing

Programmed computing has dominated information processing for the last 50 years. When information-processing functions are complex, it is appealing to consider the idea of training a system instead of programming it. This is especially true for problems in which a large number of variables must be considered as part of the decision process.

Traditional computer applications use the programmed computing approach. Solutions are devised by designing algorithms that solve the problem and then implementing them in software. The objective is to deliver quality applications that solve specific user needs on time and within budget. Programmed computing works well if there is a well understood process or set of rules for solving the given problem.

Providing solutions to novel problems when the algorithm is unknown can involve a costly and time consuming development cycle. Quality software development can require a rigorous cycle of design, validation, and incremental improvement, making it an expensive and lengthy process.

An alternative approach to programmed computing, particularly well suited to problems in areas such as sensor processing, pattern recognition, data analysis (e.g., data mining), prediction and control, is neurocomputing. Neurocomputing refers to systems that learn the relationships between data through a process of training. Neural networks are the primary information processing structure used in neurocomputing. Neurocomputing benefits include and are often measured in terms of:

• a reduction in the time it takes to solve the problem when compared to the programmed computing approach;
• a reduction in the quantity of software needed to solve the problem; and
• a convenient solution to the problem that may be too complex for programmed computing.

Typical applications of neurocomputing technology are often grouped into one of three domains: Analysis, prediction and control.

Analysis

Data analysis applications are used to discover relationships and recognize patterns within data. Data mining and pattern classification are typical analysis applications.

Data Mining

Corporate America is accumulating vast quantities of data describing their operations and storing this information in "data warehouses." Understanding the relationships in this data creates applications that can forecast sales, predict a competitor’s bid, identify new markets, and detect fraud.

Pattern Classification

Patterns in data can be detected and classified based on a sequence of input measurements. Applications include optical character recognition, sensor data classification, face recognition, trend analysis, and signal detection.

Prediction

Prediction or forecasting is the ability of the system to predict future values and outcomes based on current input values. Applications include predictive maintenance and load forecasting.

Predictive Maintenance

Based on data gathered over time on the health of a piece of machinery (including breakdowns), a predictor is used to schedule machine maintenance before the next breakdown occurs.

Load Forecasting

Historical load data is used to create a model that can forecast future load values. Successful applications include electrical power load forecasting, and telecommunications switch load forecasting.

Control

The control of machines or processes often requires high-speed computations and function inversion (the ability of the model to provide the required input given a desired output). Applications include automotive control systems and computer-controlled prostheses.

Vehicle Active Suspension Systems

Computer controlled active suspension systems allow a vehicle to adaptively adjust the firmness of the suspension system and improve handling.

Walking Aids

Computer assisted walking aids for spinal cord injured persons are used to control walking gait by detecting the user’s intended action.

An Effective Alternative to Neural Networks

Although neurocomputing benefits are many, critics have justifiably cited the "black-box" solution approach as the primary reason for not using the technology in many practical or safety-critical applications. An immensely successful neurocomputing technology that does not suffer from the "black-box" criticism is Dendronic Decisions’ Adaptive Logic Networks technology.

Back to Top

Adaptive Logic Networks Technical Overview

What it is

An Adaptive Logic Network is a form of neurocomputing capable of modeling complex non-linear systems by using piece-wise linear surfaces.

The inputs to an Adaptive Logic Network may be data from large corporate databases, observations recorded by a scientist, or real-time measurements from a manufacturing process. The outputs of an Adaptive Logic Network may be used for analysis, prediction, or real-time control of machines and processes.

Linear extrapolation techniques are often the basis of traditional data modeling tools used for prediction, but they may not be able to adequately deal with data from the real world, which is often non-linear, noisy, and contains contradictory values. Adaptive Logic Networks are a non-linear data modeling technology that overcomes these limitations.

How it works

Adaptive Logic Networks learn relationships and patterns by using a supervised learning algorithm that examines data in a training set consisting of examples of inputs and their associated outputs. During the learning phase, an Adaptive Logic Network modifies its internal structure to reflect the relationship between the inputs and the outputs in the training set. The accuracy of an Adaptive Logic Network is checked after the learning cycle is complete by using a separate set of inputs and outputs called the validation set.

Reinforcement learning is a recent addition to the algorithms developed for Adaptive Logic Networks. This type of learning is used when the desired output for a given input is not known during a sequence of actions that is taking place. During the reinforcement learning process, the only feedback given to the system is a rough indicator of performance, such as "good", "bad", "too slow", or "too fast." This type of feedback is similar to the way humans learn.

The internal structure of an Adaptive Logic Network is very simple: it is composed of one or more linear surfaces joined by simple operators. Fortunately, ordinary computers perform linear calculations and simple comparison operations very quickly. This typically eliminates the need for special hardware to solve real-world problems.

Seven Key Features of Adaptive Logic Networks

1. Safety
   Safety and mission-critical applications require that system responses be understood for all possible inputs. Since an Adaptive Logic Network model is composed of linear surfaces that are well understood mathematically, and since the Dendronic Learning Engine allows us to control how fast the output changes with changes in any input, proofs about the accuracy of an Adaptive Logic Network model for all inputs are feasible without requiring exhaustive testing.
   
2. Speed
   Adaptive Logic Networks are very fast because the evaluation of a trained network typically involves simple comparison operations and a limited number of linear surface calculations. This speedup is analogous to the alpha-beta search algorithm of games. Decision trees are very fast, and ALNs can be turned into ALN decision trees, or DTREES. (Note: these are not optimized in the present version of the DLE).
   
3. Scalability
   An Adaptive Logic Network can handle complex problems with many input variables. Instead of adapting a fixed internal architecture, an Adaptive Logic Network’s architecture can grow dynamically and efficiently in response to the complexity of the training data. The structure of an Adaptive Logic Network efficiently represents the relationship between your problem’s inputs and outputs.  The speedup under point 2 scales very well.
   
4. Broad domain applicability
   Adaptive Logic Networks have successful applications in many problem domains. An investment in Adaptive Logic Network technology can pay for itself repeatedly, greatly reducing the complexity of the tool set required to build your applications involving machine intelligence.
   
5. Embedding expert knowledge
   Learning systems often have difficulties when there is a lack of historical data for training, or data contains too much noise. Adaptive Logic Networks can compensate for these problems during the learning phase by constraining their internal structure. These constraints are often based on physical laws and rules of thumb that dictate certain relationships in the data must hold. Capturing common sense or even expert knowledge of a problem domain can compensate for sparse and noisy data, often resulting in a faster learning phase. Often rules are of the form: the greater this input is, the greater must the output be, all other inputs being equal.  (This is a special case of control over rates of change as in point 1.)
   
6. Function inversion
   The ability of a model to provide the required input given a desired output can be very useful in the real-time control of machines and processes. The process of exchanging the role of one input variable and the outputvariable, or function inversion, is facilitated by Adaptive Logic Networks because the comparison operations that combine linear surfaces in an Adaptive Logic Network preserve a mathematical property called monotonicity. Provided that the output of an ALN is monotonic in one of the inputs, the internal structure of the ALN can be rearranged so that the output of the network effectively trades places with that input.
   
7. Ease of understanding
   The learning phase of an Adaptive Logic Network is controlled by parameters directly related to the properties of the data (weights on variables correspond to rates of change of the output with respect to the inputs). Users need only be familiar with their data, not with the way the learning algorithms work. The user can concentrate on solving the problem, rather than on becoming an expert in the underlying technology.

Back to Top

   Publications

Papers Available Online

D.O.Gorodnichy, W.W.Armstrong, X. Li, Adaptive Logic Networks for Facial Feature Detection, Lecture Notes in Computer Science, Vol 1311 (Proc. of 9th Intern. Conf. on Image Analysis and Processing (ICIAP'97), Florence, Italy, Sept. 1997, Vol. II), pp. 332-339, Springer (download)

We recommend Dmitry Gorodnichy's webpages for more papers on 3Da world modeling.   The URL is the following (note different spelling of Dmitry's name!): http://www.cs.ualberta.ca/~dmitri/WorldModeling

A paper on predictive maintenance of turbine-driven compressors using ALNs (download)

A paper on rehabilitation of patients with spinal cord injury using ALNs (download)

June 1998 ART Forum Presentation (Data Mining with Adaptive Logic Networks) – http://www.peakconsulting.com/artsupp/index.htm

Tutorial on Adaptive Logic Networks in Audit Risk Assessment – http://www.peakconsulting.com/aaa.pdf

Detection of Management Fraud in Audits with Neural Nets and Internal Data – http://www.peakconsulting.com/ijis2.pdf

We recommend visiting the site of Peak Consulting for other papers on ALN applications in the areas of economic forecasting, statistical modeling in management and neural computing.  The URL is http://www.peakconsulting.com

Publications in Print

Note: publications prior to 1995 mostly refer to a  type of network which used boolean signals and nodes with AND, OR, LEFT and RIGHT gates. After 1993 the signals flowing the the network were real (realized as floating point), the leaf nodes were linear functions and the other nodes were MAXIMUM and MINIMUM operators.  The far easier credit assignment algorithm in the case of the latter nets makes the earlier ones obsolete as far as machine learning algorithms are concerned. The two types of network are easily converted into one another: one can recognize the points under a function graph while the other computes that function directly. Even in modern networks, the switch to a boolean interpretation is useful when converting an ALN to compute an inverse function (i.e. changing the output variable).

W. Armstrong, On Networks of Adaptive Boolean Logic Elements and their Application to Pattern Recognition, Proc. Fourth Princeton Conference, 1968, p. 326.

G. v. Bochmann, W. Armstrong, Properties of Boolean Functions with a Tree Decomposition, BIT 13, 1974. pp. 1-13.

W. Armstrong and G. Godbout, "Properties of Binary Trees of Flexible Elements Useful in Pattern Recognition", IEEE 1975 International Conf. on Cybernetics and Society, San Francisco, 1975, IEEE Cat. No. 75 CHO 997-7 SMC, pp. 447-449.

W. Armstrong and J. Gecsei, "Architecture of a Tree-based Image Processor", 12th Asilomar Conf. on Circuits, Systems and Computers, Pacific Grove, Calif., 1978, pp. 345-349.

W. Armstrong and J. Gecsei, "Adaptation Algorithms for Binary Tree Networks", IEEE Trans. on Systems, Man and Cybernetics, 9, 1979, pp. 276-285.

W. W. Armstrong, Andrew Dwelly, Jian-Dong Liang, Dekang Lin, Scott Reynolds, Learning and Generalization in Adaptive Logic Networks, in Artificial Neural Networks, Proc. of the 1991 Int. Conf. on Artificial Neural Networks (ICANN-91), Espoo, Finland, June 24-28, 1991,T. Kohonen, K. Makisara, O. Simula, J. Kangas, eds., North Holland, pp. 1173-1176.

W. W. Armstrong, Andrew Dwelly, Jian-Dong Liang, Dekang Lin, Scott Reynolds, Experience Using Adaptive Logic Networks, Proc. IASTED Int'l Symp. on Computers, Electronics, Communication and Control, Calgary, April 8-10, 1991, pp. 44 - 47.

Allen G. Supynuk, William W. Armstrong, Adaptive Logic Networks and Robot Control, Proc. Vision Interface Conference '92, also called AI/VI/GI '92, Vancouver B. C., May 11-15, 1992, pp. 181 - 186.

Aleksandar Kostov, Richard B. Stein, William W. Armstrong, Monroe Thomas, Evaluation of Adaptive Logic Networks for Control of Walking in Paralyzed Patients, 14th Ann. Int'l Conf. IEEE Engineering in Medicine and Biology Society, Paris, France, Oct. 29 - Nov. 1, 1992 Vol. 4, pp. 1332-1334.

Kostov, A., Popovic, D.B., Stein, R.B., and Armstrong, W.W. (1993), EMG patterns learning by Adaptive Logic Network, Proc. of the 15th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, USA, pp. 1135-1136.

W. W. Armstrong, R. B. Stein, A. Kostov, M. Thomas, P. Baudin, P. Gervais, D. Popovic, Application of adaptive logic networks and dynamics to study and control of human movement, Keynote address at the Second Int'l Symp. on 3D Analysis of Human Movement, Poitiers, France, June 30 - July 3, 1993, pp. 81 - 84.

Armstrong, W.W., Stein, R.B., Kostov, A., Thomas, M., Baudin, P., Gervais, P., and Popovic, D.B., (1993) Application of Adaptive Logic Networks and Dynamics to Study and Control of Human Movement, Proc. of the Second International Symposium on Three-Dimensional Analysis of Human Movement - satellite meeting of the International Society of Biomechanics, Poitiers, France, pp. 81-84. (keynote address by Armstrong, W.W.)

D. Popovic, R. B. Stein, K. Jovanovic, R. Dai, A. Kostov, W. Armstrong, Sensory Nerve Recording for Closed-loop Control to Restore Motor Functions, IEEE Trans. Biomed. Eng., vol. 40 no. 10 pp. 1024 - 1031, 1993.

A. Kostov, D. B. Popovic, R. B. Stein, and W. W. Armstrong, Learning of EMG-patterns by Adaptive Logic Networks, Proc. of the 15th
Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, USA, Oct. 28-31, 1993, pp. 1135-1136.

Aleksandar Kostov, Richard B. Stein, Dejan Popovic and W. W. Armstrong, Improved Methods for Control of FES for Locomotion, Proc. International Federation of Automatic Control (IFAC) Symposium on Modeling and Control in Biomedical Systems, Galveston, Texas, March 27 - 30, 1994, pp. 422 - 427.

A. Kostov,B. Andrews, R. Stein, D. Popovic, W.W. Armstrong: Machine learning in control of functional electrical stimulation for locomotion, IEEE Engineering in Medicine and Biology Society 16th Annual Conf., Baltimore MD, Nov. 1994 pp. 418 - 419.

Note: the following are based on modern ALN networks with real signals flowing between the nodes:

Kostov, A., Strange K., Stein, R.B., and Hoffer A.J., (1995), Adaptive Logic Networks in EMG-prediction from Sensory Nerve Signals Recorded in the Cat’s Forelimb During Walking, Physiology Canada, Vol. 26, No. 2, pp. 104.

M J Polak, SH Zhou, P M Rautaharju, W. W. Armstrong, B R Chaitman Adaptive logic network compared with backpropagation network in automated detection of ischemia from resting ECG, Proc. Computers in Cardiology Conf., Vienna, Austria, Sept. 10 - 13, 1995, pp 217 - 220.

W. W. Armstrong, C. Chu, and M. Thomas, Using adaptive logic networks to predict machine failure, Proc. World Congress on Neural Networks (WCNN'95) Washington DC, July 1995.

Aleksandar Kostov, Brian J. Andrews, Dejan B. Popovic, Richard B. Stein, William W. Armstrong, Machine Learning in Control of Functional Electrical Stimulation Systems for Locomotion, IEEE Trans. Biomed. Eng. vol. 42 no. 6, 1995, pp. 541 - 551.

Richard B. Stein, Aleksandar Kostov, Dejan Popovic, William W. Armstrong, and Monroe Thomas, Functional Electrical Stimulation aided locomotion controlled in real time by artificial neural networks, Can. J. Physiol. Pharmacol. 73:A29, 1995 (Abstract)

A. Kostov, R. B. Stein, W. W. Armstrong, M. Thomas, D. Popovic, Integrated control system for FES-assisted locomotion after spinal cord injury, Proc. IEEE Engineering in Medicine and Biology Society, 17th Ann. Conf., Montreal, September 20-23 1995 (CD-ROM).

W W Armstrong, Monroe M Thomas, Adaptive Logic Networks, Section C 1.8 in Handbook of Neural Computation, Emile Fiesler and Russell Beale eds., Oxford University Press, 1996, ISBN 0-7503-0312-3 (looseleaf)

W W Armstrong, Monroe M Thomas, Neural net control of an active suspension system, Section G2.1 Handbook of Neural Computation (ibid)

Kostov, A., Armstrong, W.W., Thomas M., and Stein, R.B.,  Case Study - Adaptive Logic Networks in Rehabilitation of Persons with Incomplete Spinal Cord Injury, Section G.5.1, Handbook of Neural Computation, ibid.

D.O.Gorodnichy, W.W.Armstrong, X. Li, Adaptive Logic Networks for Facial Feature Detection, Lecture Notes in Computer Science, Vol 1311, Springer Verlag, 1997, pp. 332-339. Proc. of 9th Intern. Conf. on Image Analysis and Processing ICIAP'97, Florence, Italy, Sept. 1997.

M. J. Polak, S. H. Zhou, P. M. Rautaharju, W. W. Armstrong, B. R. Chaitman, Using automated analysis of resting twelve-lead ECG to identify patients at risk of developing transient myocardial ischaemia -- an application of an adaptive logic network, Physiological Measurement 18, 1997 pp.317 - 325.

W. W. Armstrong, Reinforcement learning applied to simulated basketball balancing, Int'l. ICSC-IFAC Symposium on Neural Computation, NC'98, Vienna, Sept 23 - 25, 1998, CD-ROM ISBN 3-906454-14-2

S. Ramaswamy, T. Ono-Tesfaye, W. W. Armstrong and P. Gburzynski, Equivalent Bandwidth Characterization for Real-time CAC in ATM Networks, Journal of High Speed Networks, 1998, pp 1-25.

William W. Armstrong and Darwin Li, A new technique for reinforcment learning in control, 1998 IEEE Conf. on Systems, Man and Cybernetics, La Jolla CA, Oct. 11 -14, 1998, ISBN 0-7803-4781-1, IEEE cat. no. 98CH36218.

D.O. Gorodnichy, W.W. Armstrong, Single Camera Stereo for Mobile Robots, Proc. Vision Interface (VI'99), Trois Rivieres, Canada, May 18-21, 1999, pp. 528-535.

D.O. Gorodnichy, W.W. Armstrong, A Parametrical Alternative for Grids in Occupancy Based World Modeling, Proc. Quality Control by Artificial Vision Conference (QCAV'99), Trois Rivieres, Canada, May 18-21, 1999, pp 125 - 132.

W.W. Armstrong, B. Coghlan, D.O. Gorodnichy, Reinforcement learning for autonomous robot navigation, Proc. Int'l Joint Conf. on Neural Networks (IJCNN'99), Washington DC, July 21-23, 1999

G. Qu, J. J. R. Feddes, W. Armstrong, J. Leonard, R. Coleman, Combining and Electronic Nose with an Artificial Neural Network to Measure Odour Concentration, ASAE/CSAE-SCGR Annual Int'l Meeting, July 18-21, 1999, Toronto (not refereed).

G.Qu, J.J.R.Feddes, W.W.Armstrong, R.N.Coleman and J.J.Leonard, Measuring odor concentration with an electronic nose, Proc. Second International Conference on Air Pollution from Agricultural Operations, October 9-11, 2000, Des Moines, Iowa. pp 188-195. Library of congress card number (LCCN)00-134838. International standard book number (ISBN)1-892769-12-3. ASAE Publication 701P003.

W. W. Armstrong, D. O. Gorodnichy, Breaking Hyperplanes to fit Data with Applications to 3D World Modeling and Oil Sand Data Analysis,  Proc. ICSC Symposium on Neural Computation, NC 2000, Berlin, May 23 - 26, 2000, CD-ROM publ. by ICSC Academic Press, Int'l Comp. Sci. Conv., Canada/Switzerland ISBN 3-906454-21-5

Dmitry O. Gorodnichy, W. W. Armstrong, Neurocomputational Approach for Modeling Large Scale Environments from Range Data, ibid.

W. W. Armstrong, High speed networks that preserve continuity and accuracy, Int'l Joint Conf. on Neural Networks, Washington DC, July 14 - 19, 2001 - Special session on morphological networks, CD-ROM.

    Back to Top

Send mail to with questions or comments about this web site.
Copyright © 2003 Dendronic Decisions Limited. All rights reserved.
Date Modified: March 1, 2003.