Home > The Theory and Application of Artificial Neural Networks

The Theory and Application of Artificial Neural Networks


1  

CogNova

Technologies 

Theory and Application of Artificial Neural Networks 

with

Daniel L. Silver, PhD 

Copyright  (c), 2014

All Rights Reserved


2  

CogNova

Technologies 

Seminar Outline 

DAY 1 

  • ANN Background and  Motivation
  • Classification Systems and  Inductive Learning
  • From Biological to Artificial Neurons
  • Learning in a Simple Neuron
  • Limitations of Simple Neural Networks
  • Visualizing the Learning Process
  • Multi-layer Feed-forward ANNs
  • The Back-propagation Algorithm
 

DAY  2 

  • Generalization in ANNs
  • How to Design a Network
  • How to Train a Network
  • Mastering ANN Parameters
  • The Training Data
  • Post-Training Analysis
  • Pros and Cons of  Back-prop
  • Advanced issues and networks

 


3  

CogNova

Technologies 

ANN Background and 
Motivation


4  

CogNova

Technologies 

Background and Motivation 

  • Growth has been explosive since 1987
    • education institutions, industry, military
    • > 500 books on subject
    • > 20 journals dedicated to ANNs
    • numerous popular, industry, academic articles 
  • Truly inter-disciplinary area of study
  • No longer a  flash in the pan technology

5  

CogNova

Technologies 

Background and Motivation 

  • Computers and the Brain: A Contrast 
    • Arithmetic: 1 brain = 1/10 pocket calculator
    • Vision:   1 brain = 1000 super computers  
    • Memory of arbitrary details:    computer wins
    • Memory of real-world facts:     brain wins    
    • A computer must be programmed explicitly
    • The brain can learn by experiencing the world

6  

CogNova

Technologies 

Background and Motivation


7  

CogNova

Technologies 

Background and Motivation 

Inherent Advantages of the Brain:

           ��distributed processing and representation�� 

    • Parallel processing speeds
    • Fault tolerance
    • Graceful degradation
    • Ability to generalize
 

I 

O 

f(x) 

x


8  

CogNova

Technologies 

Background and Motivation 

History of Artificial Neural Networks

  • Creation:

    1890: William James - defined a neuronal process of learning

  • Promising Technology:

    1943: McCulloch and Pitts - earliest mathematical models

    1954: Donald Hebb and IBM research group - earliest simulations

    1958: Frank Rosenblatt -  The Perceptron

  • Disenchantment:

    1969: Minsky and Papert - perceptrons have severe limitations

  • Re-emergence:

    1985: Multi-layer nets that use back-propagation

    1986: PDP Research Group - multi-disciplined approach

9  

CogNova

Technologies 

ANN application areas ...

  • Science and medicine: modeling, prediction, 

                                      diagnosis, pattern recognition

  • Manufacturing: process modeling and analysis
  • Marketing and Sales:  analysis, classification,

                                              customer targeting

  • Finance:  portfolio trading, investment support
  • Banking & Insurance:  credit and policy approval
  • Security:   bomb,  iceberg,  fraud detection
  • Engineering:  dynamic load schedding, pattern

                                           recognition 

Background and Motivation


10  

CogNova

Technologies 

Classification Systems 
and Inductive Learning


11  

CogNova

Technologies 

Classification Systems  
and Inductive Learning
 

Basic Framework for Inductive Learning 

Inductive

Learning System 

Environment 

Training

Examples 

Testing

Examples 

Induced

Model of

Classifier 

Output Classification 

(x, f(x)) 

(x, h(x)) 

h(x) = f(x)? 

A problem of representation and

search for the best hypothesis, h(x). 

~


12  

CogNova

Technologies 

Classification Systems  
and Inductive Learning
 

Vector Representation & Discriminate Functions 

x 

x 

Height 

Age 

2 

1 

* 

o 

* 

o 

Class Clusters 

��Input or Attribute

Space�� 

* 

* 

* 

o 

o 

o 

A 

B


13  

CogNova

Technologies 

Classification Systems  
and Inductive Learning
 

Vector Representation & Discriminate Functions 

x 

x 

Height 

Age 

2 

1 

* 

o 

* 

o 

* 

* 

* 

o 

o 

o 

A 

B 

Linear Discriminate

Function 

f(X)=(x1,x2) =

w0+w1x1+w2x2=0

or WX = 0

f(x1,x2) > 0 => A

f(x1,x2) < 0 => B 

-w0/w2

 


14  

CogNova

Technologies 

Classification Systems  
and Inductive Learning
 

  • f(X) = WX =0 will discriminate  class A from B,
  • BUT ... we do not know the appropriate values for :

                      w0, w1, w2 
 
 

 


15  

CogNova

Technologies 

Classification Systems  
and Inductive Learning
 

We will consider one family of neural network classifiers:

  • continuous valued input
  • feed-forward
  • supervised learning
  • global error

16  

CogNova

Technologies 

From Biological to Artificial Neurons


17  

CogNova

Technologies 

From Biological to Artificial Neurons 

The Neuron - A Biological Information Processor

  • dentrites -   the receivers
  • soma -   neuron cell body (sums input signals)
  • axon  -   the transmitter
  • synapse -   point of transmission
  • neuron activates after a certain threshold is met

Learning occurs via electro-chemical changes in effectiveness of synaptic junction


18  

CogNova

Technologies 

From Biological to Artificial Neurons 

An Artificial Neuron - The Perceptron

  • simulated on hardware or by software
  • input connections -   the receivers
  • node, unit, or PE simulates neuron body
  • output connection -   the transmitter
  • activation function employs a threshold or bias
  • connection weights act as synaptic junctions

Learning occurs via changes in value of the connection weights.


19  

CogNova

Technologies 

From Biological to Artificial Neurons 

An Artificial Neuron - The Perceptron

  • Basic function of neuron is to sum inputs, and produce output given sum is greater than threshold
  • ANN node produces an output as follows:

    1.  Multiplies each component of the input pattern by the weight of its connection

    2.  Sums all weighted inputs and subtracts the threshold value => total weighted input

    3.  Transforms the total weighted input into the output using the activation function


20  

CogNova

Technologies 

From Biological to Artificial Neurons 

Hidden Nodes 

Output Nodes 

Input Nodes 

I1 

I2 

I3 

I4 

O1 

O2 

��Distributed processing

and representation�� 

3-Layer Network

has

2 active layers


21  

CogNova

Technologies 

From Biological to Artificial Neurons 

Behaviour of an artificial neural network to any particular input depends upon: 

  • structure of each node (activation function)
  • structure of the network (architecture)
  • weights on each of the connections

                               ....  these must be learned !


22  

CogNova

Technologies 

Learning in a Simple Neuron


23  

CogNova

Technologies 

Learning in a Simple Neuron  

H = {W|W R(n+1)} 

x1 

x2 

y 

0   0    0

0   1    0

1   0    0

1   1    1 

x1 

x2 

x0=1 

w1 

w0=  

w2 

Fries 

Burger 

where f(a) is the

step function, such

that:   f(a)=1, a > 0

          f(a)=0, a <= 0 

��Full Meal Deal��


24  

CogNova

Technologies 

Learning in a Simple Neuron  

Perceptron Learning Algorithm:

1. Initialize weights

2. Present a pattern and target output

3. Compute output :

4. Update weights :

Repeat starting at 2 until acceptable level of error


25  

CogNova

Technologies 

Learning in a Simple Neuron  

Widrow-Hoff  or Delta Rule     for Weight Modification 
 

Where:

h   = learning rate (o < h <= 1),     typically set = 0.1

d = error signal  =  desired output - network output

                   =   t - y 

;


26  

CogNova

Technologies 

Learning in a Simple Neuron  

Perceptron Learning - A Walk Through 

  • The PERCEPT.XLS table represents 4 iterations through training data for ��full meal deal�� network
  • On-line weight updates
  • Varying learning rate, h , will vary training time

27  

CogNova

Technologies 

TUTORIAL #1 

  • Your ANN software package: A Primer
  • Develop and train a simple neural network to learn the  OR  function 

28  

CogNova

Technologies 

Limitations of Simple Neural Networks


29  

CogNova

Technologies 

Limitations of Simple Neural Networks 

What is a Perceptron doing when it learns?

  • We will see it is often good to visualize network activity
  • A discriminate function is generated
  • Has the power to map input patterns to output class values
  • For 3-dimensional input, must visualize 3-D space and 2-D hyper-planes

30  

CogNova

Technologies 

EXAMPLE 

Logical OR

Function 

x1 x2  y

0 0 0

0 1 1

  1. 0 1

1 1 1 

x2 

x1 

0,0 

0,1 

1,0 

1,1 

y = f(w0+w1x1+w2x2) 

What is an artificial neuron doing when it learns? 

Simple

Neural Network


31  

CogNova

Technologies 

Limitations of Simple Neural Networks 

The Limitations of Perceptrons

(Minsky and Papert, 1969)

  • Able to form only linear discriminate functions; i.e.  classes which can be divided by a line or hyper-plane
  • Most functions are more complex; i.e. they are non-linear or not linearly separable
  • This crippled research in neural net theory for 15 years ....

32  

CogNova

Technologies 

EXAMPLE 

Logical XOR

Function 

x1 x2  y

0 0 0

0 1 1

  1. 0 1

1 1 0 

0,0 

0,1 

1,0 

1,1 

Two neurons are need!  Their combined results

can produce good classification. 

Hidden layer

of neurons 

Multi-layer

Neural Network 

x2 

x1


33  

CogNova

Technologies 

EXAMPLE 

More complex multi-layer networks are needed

to solve more difficult problems. 


B


34  

CogNova

Technologies 

TUTORIAL #2 
 

  • Develop and train a simple neural network to learn the  XOR  function
  • Also see: 

    http://www.neuro.sfc.keio.ac.jp/~masato/jv/sl/BP.html

 


35  

CogNova

Technologies 

Multi-layer Feed-forward ANNs


36  

CogNova

Technologies 

Multi-layer Feed-forward ANNs 

Over the 15 years (1969-1984) some research continued ...

  • hidden layer of nodes allowed combinations of linear functions
  • non-linear activation functions  displayed properties closer to real neurons:
    • output varies continuously but not linearly
    • differentiable .... sigmoid

        non-linear ANN classifier was possible


37  

CogNova

Technologies 

Multi-layer Feed-forward ANNs 

  • However ... there was no learning algorithm to adjust the weights of a multi-layer network  - weights had to be set by hand.
  • How could the weights below the hidden layer be updated? 

38  

CogNova

Technologies 

Visualizing Network Behaviour


39  

CogNova

Technologies 

Visualizing Network Behaviour 

  • Pattern Space
  • Weight Space 
  • Visualizing the process of learning 
    • function surface in weight space
    • error surface in weight space
 

x1 

x2 

w0 

w2 

w1


40  

CogNova

Technologies 

The Back-propagation Algorithm


41  

CogNova

Technologies 

The Back-propagation Algorithm 

  • 1986: the solution to multi-layer ANN weight update rediscovered
  • Conceptually simple - the global error is backward propagated to network nodes, weights are modified proportional to their contribution
  • Most important ANN learning algorithm
  • Become known as back-propagation because the error is send back through the network to correct all weights

42  

CogNova

Technologies 

The Back-propagation Algorithm 

  • Like the Perceptron - calculation of error is based on difference between target and actual output:
  • However in BP it is the rate of change of the error which is the important feedback through the network 

          generalized delta rule

  • Relies on the sigmoid activation function for communication

43  

CogNova

Technologies 

The Back-propagation Algorithm 

Objective:   compute             for all

Definitions:

            =  weight from node i to node j

            =  totaled weighted input of node

        

            = output of node    

            = error for 1 pattern over all output         nodes 


44  

CogNova

Technologies 

The Back-propagation Algorithm 

Objective:   compute             for all

Four step process:

1. Compute how fast  error changes as output of node j is changed

2. Compute how fast error changes as total input to node j is changed

3. Compute how fast error changes as weight         coming into node j is changed

4. Compute how fast error changes as output of node i in previous layer is changed


45  

CogNova

Technologies 

The Back-propagation Algorithm 

On-Line algorithm:

1. Initialize weights

2. Present a pattern and target output

3. Compute output :

4. Update weights :

                      where 

Repeat starting at 2 until acceptable level of error


46  

CogNova

Technologies 

The Back-propagation Algorithm 

Where: 

For output nodes: 

For hidden nodes: 

 


47  

CogNova

Technologies 

The Back-propagation Algorithm 

Visualizing the bp learning process:

The bp algorithm performs a gradient descent in weights space toward a minimum level of error using a fixed step size or learning rate 

The gradient is given by :

= rate at which error changes as weights change


48  

CogNova

Technologies 

The Back-propagation Algorithm 

Momentum Descent:

  • Minimization can be speed-up if an additional term is added to the update equation:

        where:

  • Thus:
      • Augments the effective learning rate      to vary the  amount a weight is updated
      • Analogous to momentum of a ball - maintains direction
      • Rolls through small local minima
      • Increases weight upadte when on stable gradient

49  

CogNova

Technologies 

The Back-propagation Algorithm 

Line Search Techniques:

  • Steepest and momentum descent use only gradient of error surface
  • More advanced techniques explore the weight space using various heuristics
  • Most common is to search ahead in the direction defined by the gradient

50  

CogNova

Technologies 

The Back-propagation Algorithm 

On-line vs. Batch algorithms:

  • Batch (or cumulative) method reviews a set of training examples known as an epoch and computes global error:
  • Weight updates are based on this cumulative error signal 
  • On-line more stochastic and typically a little more accurate, batch more efficient

51  

CogNova

Technologies 

The Back-propagation Algorithm 

Several Interesting Questions:

  • What is BP��s inductive bias?
  • Can BP get stuck in local minimum?
  • How does learning time scale with size of the network & number of training examples?
  • Is it biologically plausible?
  • Do we have to use the sigmoid activation function?
  • How well does a trained network generalize to unseen test cases?  

52  

CogNova

Technologies 

TUTORIAL #3 
 

  • The XOR  function revisited
  • Software package tutorial : 

                          Electric Cost Prediction

53  

CogNova

Technologies 

Generalization


54  

CogNova

Technologies 

Generalization 

  • The objective of learning is to achieve good generalization to new cases, otherwise just use a look-up table.
  • Generalization can be defined as a mathematical interpolation or regression over a set of training points:
 

f(x) 

x


55  

CogNova

Technologies 

Generalization 

An Example: Computing Parity 

                                  Can it learn from       m examples to       generalize to all       2^n possibilities? 

>0 

>1 

>2 

Parity bit value 

(n+1)^2

weights 

n bits of input 

2^n possible examples 

+1 

-1 

+1


56  

CogNova

Technologies 

Generalization 

Fraction of cases used during training 

Test

Error 

100% 

0 

.25 

.50 

.75 

1.0 

Network test of 10-bit parity

(Denker et. al., 1987) 

When number of training cases,

m >> number of weights, then

generalization occurs.


57  

CogNova

Technologies 

Generalization 

A Probabilistic Guarantee

N = # hidden nodes m = # training cases

W = # weights        = error tolerance (<  1/8)

Network will generalize with 95% confidence if:

    1. Error on training set <

    2.  

Based on PAC theory => provides a good rule of practice.


58  

CogNova

Technologies 

Generalization 

Consider 20-bit parity problem:

  • 20-20-1 net has 441 weights
  • For 95% confidence that net will predict with                  , we need

                                     training examples 

  • Not bad considering

59  

CogNova

Technologies 

Generalization 

Training Sample & Network Complexity

                     Based on                 : 
 
 
 

W -  to reduced size

      of training sample 

W -  to supply freedom

      to construct desired function 

Optimum W=> Optimum #

            Hidden Nodes


60  

CogNova

Technologies 

Generalization 

How can we control number of effective weights?

  • Manually or automatically select optimum number of hidden nodes and connections
  • Prevent over-fitting = over-training
  • Add a weight-cost term to the bp error equation

61  

CogNova

Technologies 

Generalization 

Over-Training

  • Is the equivalent of over-fitting a set of data points to a curve which is too complex
  • Occam��s Razor (1300s) :  ��plurality   should not be assumed without necessity��
  • The simplest model which explains the majority of the data is usually the best

62  

CogNova

Technologies 

Generalization 

Preventing Over-training:

  • Use a separate test or tuning set of examples
  • Monitor error on the test set as network trains
  • Stop network training just prior to over-fit error occurring  - early stopping  or  tuning
  • Number of effective weights is reduced
  • Most new systems have automated early stopping methods

63  

CogNova

Technologies 

Generalization 

Weight Decay: an automated method of     effective weight control

  • Adjust the bp error function to penalize the growth of unnecessary weights:
 
 

     where:      = weight -cost parameter

       is decayed by an amount proportional to its magnitude;   those not reinforced => 0


64  

CogNova

Technologies 

TUTORIAL #4 
 

  • Generalization: Develop and train a BP network to learn the  OVT  function

65  

CogNova

Technologies 

Network Design & Training


66  

CogNova

Technologies 

Network Design & Training Issues  

Design:

  • Architecture of network
  • Structure of artificial neurons
  • Learning rules

Training:

  • Ensuring optimum training
  • Learning parameters
  • Data preparation
  • and more ....

67  

CogNova

Technologies 

Network Design


68  

CogNova

Technologies 

Network Design 

Architecture of the network: How many nodes?

  • Determines number of network weights
  • How many layers?
  • How many nodes per layer?

    Input Layer        Hidden Layer       Output Layer 

  • Automated methods:
    • augmentation (cascade correlation)
    • weight pruning and elimination

69  

CogNova

Technologies 

Network Design 

Architecture of the network:  Connectivity?

  • Concept of model or hypothesis space
  • Constraining the number of hypotheses:
    • selective connectivity
    • shared weights
    • recursive connections

70  

CogNova

Technologies 

Network Design 

Structure of artificial neuron nodes

  • Choice of input integration:
    • summed, squared and summed
    • multiplied
  • Choice of activation (transfer) function:
    • sigmoid (logistic)
    • hyperbolic tangent
    • Guassian
    • linear
    • soft-max

71  

CogNova

Technologies 

Network Design 

Selecting a Learning Rule

  • Generalized delta rule (steepest descent)
  • Momentum descent
  • Advanced weight space search techniques
  • Global Error function can also vary

          - normal       - quadratic         - cubic

72  

CogNova

Technologies 

Network Training 
 

 


73  

CogNova

Technologies 

Network Training 

How do you ensure that a network has been well trained?

  • Objective:  To achieve good generalization

                    accuracy on new examples/cases

  • Establish a maximum acceptable error rate
  • Train the network using a validation test set to tune it
  • Validate the trained network against a separate test set which is usually referred to as a production test set

74  

CogNova

Technologies 

Network Training 

Available  Examples 

Training

Set 

Production

Set 

Approach #1:     Large Sample

When the amount of available data is large ... 

70% 

30% 

Used to develop one ANN model 

Compute

Test error 

Divide randomly 

Generalization error

= test error 

Test

Set


75  

CogNova

Technologies 

Network Training 

Available Examples 

Training

Set 

Pro.

Set 

Approach #2:   Cross-validation

When the amount of available data is small ... 

10% 

90% 

Repeat 10

times 

Used to develop 10 different ANN models 

Accumulate

test errors 

Generalization error

determined by mean

test error and stddev 

Test

Set


76  

CogNova

Technologies 

Network Training 

How do you select between two ANN designs ? 

  • A statistical test of hypothesis is required to ensure that a significant difference exists between the error rates of two ANN models
  • If Large Sample method has been used then apply McNemar��s test*
  • If Cross-validation then use a paired t test for difference of two proportions
 

*We assume a classification problem, if this is function

approximation then use paired t test for difference of  means


77  

CogNova

Technologies 

Network Training 

Mastering ANN Parameters

                                   Typical           Range

learning rate -            0.1              0.01 - 0.99

momentum -              0.8              0.1 - 0.9

weight-cost -              0.1              0.001 - 0.5

Fine tuning :  - adjust individual parameters at each node and/or connection weight

    • automatic adjustment during training

78  

CogNova

Technologies 

Network Training 

Network weight initialization

  • Random initial values  +/- some range
  • Smaller weight values for nodes with many incoming connections
  • Rule of thumb:   initial weight range should be approximately
 

    coming into a node


79  

CogNova

Technologies 

Network Training 

Typical Problems During Training 

E 

# iter 

E 

# iter 

E 

# iter 

Would like: 

But

sometimes: 

Steady, rapid decline

in total error 

Seldom a local minimum -  reduce learning or momentum parameter 

Reduce learning parms.

- may indicate data is not learnable


80  

CogNova

Technologies 

Data Preparation 
 

 


81  

CogNova

Technologies 

Data Preparation 

Garbage in      Garbage out

  • The quality of results relates directly to quality of the data
  • 50%-70% of ANN development time will be spent on data preparation
  • The three steps of data preparation:
    • Consolidation and Cleaning
    • Selection and Preprocessing
    • Transformation and Encoding

82  

CogNova

Technologies 

Data Preparation 

Data Types and ANNs

  • Four basic data types:
    • nominal discrete symbolic (blue,red,green)
    • ordinal discrete ranking (1st, 2nd, 3rd)
    • interval measurable numeric (-5, 3, 24)
    • continuous numeric (0.23, -45.2, 500.43)
  • bp ANNs accept only continuous numeric values (typically 0 - 1 range)

83  

CogNova

Technologies 

Data Preparation 

Consolidation and Cleaning

  • Determine appropriate input attributes
  • Consolidate data into working database
  • Eliminate or estimate missing values
  • Remove outliers (obvious exceptions)
  • Determine prior probabilities of categories and deal with volume bias

84  

CogNova

Technologies 

Data Preparation 

Selection and Preprocessing

  • Select examples         random sampling

          Consider number of training examples?

  • Reduce attribute dimensionality
    • remove redundant and/or correlating attributes
    • combine attributes (sum, multiply, difference)
  • Reduce attribute value ranges
    • group symbolic discrete values
    • quantize continuous numeric values

85  

CogNova

Technologies 

Data Preparation 

Transformation and Encoding

Nominal or Ordinal values

  • Transform to discrete numeric values
  • Encode the value 4 as follows:
    • one-of-N code (0 1 0 0 0) - five inputs
    • thermometer code ( 1 1 1 1 0) - five inputs
    • real value (0.4)*  - one input if ordinal
  • Consider relationship between values
    • (single, married, divorcevs.  (youth, adult, senior)
 

* Target values should be 0.1 - 0.9 , not 0.0 - 1.0 range


86  

CogNova

Technologies 

Data Preparation 

Transformation and Encoding

Interval or continuous numeric values

  • De-correlate example attributes via normalization of values:
    • Euclidean:   n = x/sqrt(sum of all x^2)
    • Percentage:  n =  x/(sum of all x)
    • Variance based:  n = (x - (mean of all x))/variance
  • Scale values using a linear transform if data is uniformly distributed or use non-linear (log, power) if skewed distribution

87  

CogNova

Technologies 

Data Preparation 

Transformation and Encoding

Interval or continuous numeric values

Encode the value 1.6 as:

    • Single real-valued number (0.16)* - OK!
    • Bits of a binary number (010000) - BAD!
    • one-of-N quantized intervals (0 1 0 0 0)

       - NOT GREAT! - discontinuities

    • distributed (fuzzy) overlapping intervals 

      ( 0.3 0.8 0.1 0.0 0.0) - BEST! 

* Target values should be 0.1 - 0.9 , not 0.0 - 1.0 range


88  

CogNova

Technologies 

TUTORIAL #5 
 

  • Develop and train a BP network on real-world data
  • Also see slides covering Mitchell��s Face Recognition example

89  

CogNova

Technologies 

Post-Training Analysis


90  

CogNova

Technologies 

Post-Training Analysis 

Examining the neural net model:

  • Visualizing the constructed model
  • Detailed network analysis

Sensitivity analysis of input attributes:

  • Analytical techniques
  • Attribute elimination

91  

CogNova

Technologies 

Post-Training Analysis 

Visualizing the Constructed Model

  • Graphical tools can be used to display output response as selected input variables are changed
 

Response 

Size 

Temp


92  

CogNova

Technologies 

Post-Training Analysis 

Detailed network analysis

  • Hidden nodes form internal representation
  • Manual analysis of weight values often difficult - graphics very helpful
  • Conversion to equation, executable code
  • Automated ANN to symbolic logic conversion is a hot area of research

93  

CogNova

Technologies 

Post-Training Analysis 

Sensitivity analysis of input attributes

  • Analytical techniques
    • factor analysis
    • network weight analysis
  • Feature (attribute) elimination
    • forward feature elimination
    • backward feature elimination

94  

CogNova

Technologies 

The ANN Application Development Process 

Guidelines for using neural networks

1. Try the best existing method first

2. Get a big training set

3. Try a net without hidden units

4. Use a sensible coding for input variables

5. Consider methods of constraining network

6. Use a test set to prevent over-training

7. Determine confidence in generalization through cross-validation


95  

CogNova

Technologies 

Example Applications 

  • Pattern Recognition (reading zip codes)
  • Signal Filtering (reduction of radio noise)
  • Data Segmentation (detection of seismic onsets)
  • Data Compression (TV image transmission)
  • Database Mining (marketing, finance analysis)
  • Adaptive Control (vehicle guidance)

96  

CogNova

Technologies 

Pros and Cons of Back-Prop


97  

CogNova

Technologies 

Pros and Cons of Back-Prop  

Cons:

  • Local minimum - but not generally a concern
  • Seems biologically implausible
  • Space and time complexity:          lengthy training times
  • It��s a black box!  I can��t see how it��s making decisions?
  • Best suited for supervised learning
  • Works poorly on dense data with few input variables

98  

CogNova

Technologies 

Pros and Cons of Back-Prop  

Pros:

  • Proven training method for multi-layer nets
  • Able to learn any arbitrary function (XOR)
  • Most useful for non-linear mappings
  • Works well with noisy data
  • Generalizes well given sufficient examples
  • Rapid recognition speed
  • Has inspired many new learning algorithms

99  

CogNova

Technologies 

Other Networks and  
Advanced Issues


100  

CogNova

Technologies 

Other Networks and Advanced Issues 

  • Variations in feed-forward architecture
    • jump connections to output nodes
    • hidden nodes that vary in structure
  • Recurrent networks with feedback connections
  • Probabilistic networks
  • General Regression networks
  • Unsupervised self-organizing networks

101  

CogNova

Technologies 

THE END 
 
 
Thanks for your participation!


Set Home | Add to Favorites

All Rights Reserved Powered by Free Document Search and Download

Copyright © 2011
This site does not host pdf,doc,ppt,xls,rtf,txt files all document are the property of their respective owners. complaint#nuokui.com
TOP