Home > Hopfield NNets N. Laskaris Professor John Hopfield The Howard A. Prior Professor of Molecular Biology Dept. of Molecula

Hopfield NNets

N. Laskaris

**Professor
John Hopfield **

**The Howard
A. Prior Professor of Molecular Biology**** **

**Dept. of
Molecular Biology **

**Computational
Neurobiology; Biophysics **

**Princeton
University**** **

**The **__physicist__** Hopfield showed
that models
of physical systems could be used
to solve computational problems**

**Such systems
could be implemented
in hardware by combining
standard components
such as capacitors and resistors. **

**The importance
of the Hopfield nets
in practical application is limited **

**due to theoretical
limitations of the structure,**

**but, in
some cases, **

**
they may form interesting models.**

**Usually
employed in binary-logic tasks :
e.g. pattern completion and association**

**The concept**

**In the beginning
of 80s
Hopfield published two scientific papers,
which attracted much interest. **

**This was
the starting point of the new era
of neural networks, which continues today ******

**(1982):**** ****����Neural
networks and physical systems with emergent collective computational
abilities����.**** ****Proceedings of the
National Academy of Sciences, pp. 2554-2558****. **

**(1984):**** ****����Neurons
with graded response have collective computational properties like those
of two-state neurons����.**** ****Proceedings of the
National Academy of Sciences, pp. 81:3088-3092**

*����The
dynamics
of brain computation��*

**How
is one to understand
the incredible effectiveness of a brain **

**in
tasks such as recognizing
a particular face in a complex scene?**

**The core
question :**

**Simple models
of the dynamics
of neural circuits are described
that have **__collective__** dynamical properties****. **

**These can
be exploited
in recognizing sensory patterns.**** **

**Using
these ****collective**** properties
in processing information **

**is
effective in that**** **

**it
exploits the**** ****spontaneous**** ****properties
of nerve cells and circuits **

**to
produce robust computation.**

**Like all
computers, **

**a brain
is a dynamical system
that carries out its computations
by the change of its 'state' with time.**

**Associative
memory, **

**logic
and inference, **

**recognizing
an odor or a chess position, **

**parsing
the world into objects, **

**and
generating appropriate sequences of locomotor muscle commands **

**are
all describable **

**as ****computation****. **

**His
research focuses **

**on
understanding **

**how
the neural circuits of the brain **

**produce
such powerful and complex****
computations.**

**J. Hopfield��s
quest **

**While the
brain is totally unlike modern computers, **

**much of
what it does can be described as **__computation__**.**

**However,
olfaction allows remote sensing, **

**and
much more complex computations**** **

**involving
wind direction **

**and
fluctuating mixtures of odors **

**must
be described to account for the ability **

**of
homing pigeons or slugs to navigate **

**through
the use of odors. **

**Hopfield
has been studying
how such computations might be performed
by ****the
known neural circuitry
of the olfactory bulb
and prepiriform cortex****
of mammals
or the analogous circuits of simpler animals.**

**Olfaction**

**The simplest
problem in olfaction
is simply identifying a known odor. **

**Any computer
does its computation
by its changes in internal state.**** **

**In neurobiology, **

**
the change of potentials of neurons **

**(****and changes in the
strengths of the synapses****)
with time is what performs the computations.**** **

**Dynamical
systems**

**Systems
of differential equations
can represent these aspects of neurobiology.**** **

**He seeks
to understand some aspects of neurobiological computation **

**through
studying the behavior of equations
modeling the time-evolution of neural activity.**

**Action potential
computation**

**For much
of neurobiology, **

**information
is represented
by the paradigm of ����***firing
rates***����, **

**i.e. ****information is represented **

**by the rate
of generation of action potential spikes, **

**and the
exact timing of these spikes is unimportant****.**

**Action potential
computation**

**Since
action potentials **

**last
only about a millisecond, **

**the
use of action potential timing **

**seems
a powerful potential means of neural computation****.**

**Action potential
computation**

**There
are cases,
for example**** ****the binaural auditory
determination of the location of a sound source, **

**where
information is encoded
in the ****timing**** of action potentials.**

**Identifying
words in natural speech is a difficult computational task which brains
can easily do. **

**They use
this task as a test-bed
for thinking about **

**the computational
abilities of neural networks **

**and neuromorphic
ideas**

**Speech**

**Simple
(e.g. binary-logic ) neurons
are coupled in a system
with recurrent signal flow**** **

**A ****2****-neurons Hopfield
network
of ****continuous
states****
characterized by ****2
stable states**** **

**1**^{st}**
Example**

Contour-plot

**A ****3-****neurons Hopfield
network of ****2**^{3}**=8
states****
characterized by ****2
stable states**** **

**2**^{nd}**
Example**

**W**_{ij }
**= W**_{ji }

**The behavior
of such a dynamical system
is fully determined by the synaptic weights**** **

**And
can be thought of as
an Energy minimization process**** **

**3**^{rd}**
Example**

**Hopfield
Nets are ****fully
connected****, ****symmetrically-weighted ****networks
that extended the ideas of ***linear
associative memories***
by adding cyclic connections .**

**Note: no
self-feedback !**

**Regarding
training a Hopfield net
as a ****content-addressable
memory**** **

**the ****outer-product**** rule for storing
patterns is used**

**After the
��****teaching-stage****��,
in which the weights are defined,
the initial state of the network is set (****input
pattern****)
and a simple recurrent rule is iterated
till convergence to a stable state (****output
pattern****)**** **

**Operation
of the network**

**There are
two main modes of operation:**

** ****Synchronous ****vs.**** ****Asynchronous**** ****updating **

**Hebbian
Learning**

**Probe pattern **

**Dynamical
evolution**

**A Simple
Example**

**Step_1****.**** ****Design a network****
with memorized patterns (vectors) ****[
1, -1, 1 ] ****
& ****[
-1, 1, -1 ]**** **

**There are
8 different states
that can be reached by the net
and therefore can be used as its initial state**

**#1: y**_{1}

**#2: y**_{2}

**#3: y**_{3}

**Step_2.
Initialization**

**Step_3.
Iterate till convergence**

**- Synchronous
Updating -**** **

**3
different examples
of the net��s flow**

**It converges
immediately**

**Schematic
diagram of all the dynamical trajectories
that correspond to the designed net. **

**Stored pattern**

**Step_3.
Iterate till convergence**

**- Synchronous
Updating -**** **

**Or****
Step_3. Iterate till convergence**

**- Asynchronous
Updating -**** **

**Each time,
select one neuron
at random
and update its state
with the previous rule**

**and the
–usual- convention
that if the total input to that neuron is 0
its state remains unchanged**

**Explanation
of the convergence**

**There is
an energy function related
with each state of the Hopfield network**

**E( [y**_{1}**,
y**_{2}**, ��, y**_{n}**]**^{T}**
) = -�� �� ****w**_{ij}** y**_{i}**
y**_{j}

**where [y**_{1}**,
y**_{2}**, ��, y**_{n}**]**^{T}** **

**
is the vector of neurons�� output, **

**w**_{ij}** ****is the weight from
neuron j to neuron i****,
****and
the double sum is over i and j.**

**The corresponding
dynamical system evolves toward states of lower Energy**

**States of
lowest energy correspond to attractors
of Hopfield-net dynamics**

**E(**** [y**_{1}**,
y**_{2}**, ��, y**_{n}**]**^{T}** ****)**** =
= -�� �� ****w**_{ij}** y**_{i}**
y**_{j}

**Attractor-state**

**Capacity of
the Hopfield memory**

**When this
is found,
the corresponding pattern of activation is outputted**** **

**In short,
while ****training**** the net
(****via
the outer-product rule****)
we��re storing patterns by posing different
attractors in the state-space of the system. **

**While ****operating****,
the net searches the closest attractor.**

**How many patterns
we can store in a Hopfield-net ?**** **

****** ****0.15 N,**** ****N: # neurons**

**A
simple
Pattern Recognition
Example**

**Computer
Experimentation**

** **

**Class-project**

**Stored Patterns
(binary images)**

**Perfect
Recall-
Image Restoration**

**Erroneous
Recall**

**Irrelevant
results **

__Note:__
explain
the ��negatives�� ��.

**The **__continuous__** Hopfield-Net
as optimization machinery**** **

*��**Simple "Neural"
Optimization Networks:
An A/D Converter, Signal Decision Circuit,
and a Linear Programming Circuit**��*

**[ Tank and
Hopfield ; **

**IEEE Trans.
Circuits Syst. 1986; 33: 533-541.]:**

**Hopfield
modified his network
so as to work with continuous activation and**** **

**-by adopting
a dynamical-systems approach-**** **

**showed that
the resulting system is characterized**** ****by ****a Lyaponov-function****
who termed it ��****Computational-Energy****��**** ****& which can
be used ****to
tailor****
the net for specific optimizations**

** T**_{ij}**=T**_{ji}**
�ʦ��� T**_{ij}**=0 **

**The system
of coupled differential equation
describing the operation of continuous Hopfield net**

****** The Computational
Energy**** **

**Weights: ****W**_{ij }
**�� T**_{ij}

_{ }

**Biases: ****I**_{i}_{ }** **

**Neuronal
outputs: Y**_{i}**
�� V**_{i }

**When Hopfield
nets are used for function optimization, the objective function ***F*** to be minimized
is written as energy function in the form of computational energy ***E*** .**

**The comparison
between****
E**

**The actual
advantage of doing this
is that the Hopfield-net
has a direct hardware implementation
that enables even a ****VLSI-integration ****
of the algorithm performing the optimization task **

**An example: ***��Dominant-Mode
Clustering��*** **

**Given a
set of ****N**** vectors {X**_{i}**}
define the ****k**** among them
that form the most compact cluster {Z**_{i}**}**

**The objective
function F can be written easily
in the form of computational energy E**

**With each
pattern X**_{i}** we associate a neuron
in the Hopfield network ( i.e. #neurons = ****N**** ).**** **

**The synaptic
weights are the pairwise-distances (*2)**** **

**If its activation
is ��1�� when the net will converge
the corresponding pattern will be included in the cluster.**** **

**There��s
an additional Constraint
so as ****k**** ****neurons are ��on�� **

**A classical
example: ***��The Travelling
Salesman Problem��*** **

**Coding a
possible route as a combination
of neurons�� firings **

**The principle**

**5********3 ********4 ********1 ********2 ********5**

**|5-3|+|3-4|+|4-1|+|1-2|+|2-5|**

*The problem
:*

*The idea
:*

**An example ***from clinical
Encephalography *** **

����*Hopfield Neural Nets *

*
for monitoring Evoked Potential Signals**����*

**[ Electroenc.
Clin. Neuroph. 1997;104(2) ]**

*The solution
:*

**N. Laskaris
et al.**

** ****The Boltzmann
Machine**** **

**Improving
Hopfield nets by ****simulating
annealing****
and adopting
more complex topologies**

(430 – 355)
��.X.** **

*����ς
�ʦ˦�ί�Ҧ� �˦Ϧɦ�ό�� �Ŧ�ώ . . . .*

*
. . . . . . . . . . . . . .*

*. . .
. ��ά�ЦϦɦ�ς ά�˦˦�ς,
ί�Ҧ�ς �Ȧ� �ҦԦ̦Ц˦Ǧ�ώ�ҦŦ�
ό�Ҧ� �ĦŦ� �̦�ό�ѦŦҦ� �ͦ� �Ϧ˦Ϧʦ˦Ǧ�ώ�Ҧ�**��*

**- ���Ŧ̦ɦҦӦϦ�έ�ͦ�ς
�� ���ԦѦ��ʦ�ύ�Ҧɦ�ς
1�� έ�Ӧ�ς �Ӧ�ς 105��ς ���˦Ԧ̦Ц�ά�Ħ�ς**

**����������������**

**(1979-**__1982__**) **

*Hopfield-nets*

*PNAS** *

**(**__1982__**) **

*����
���� �Ц��ɦĦ�ά �ҦӦǦ� ���ŦѦ�ί�Ħ�
��ί�ͦ��� �� ��ό�ͦ� �ҦϦ� ���˦�ί�Ħ� ....����*

**A
Very Last Comment
on Brain-Mind-Intelligence-Life-Happiness**** **

*How
I Became Stupid*

**
by **

**Martin
Page
**

**Penguin
Books, 2004, 160 pp.
ISBN: 0-14-200495-2
**

**In HOW I
BECAME STUPID, **

**The 25-year-old
Antoine concludes**

**����to
think is to suffer����****, **

**a twist
on the familiar assertion of***
Descartes***. **

**For Antoine, ****intelligence is
the source of unhappiness****. **

**He embarks
on a series of hilarious strategies
to make himself
****stupid**** ****and possibly happy**** **

*Animals
that Abandon
their Brains *

**Dr. Jun
Aruga **

**Laboratory
for Comparative Neurogenesis
**

**
A ��primitive but successful�� animal**

**Oxycomanthus
japonicus**

**There is
astonishing diversity in the nervous systems of animals, and the variation
between species is remarkable****. **

**From the
basic, distributed nervous systems of jellyfish and sea anemones to
the centralized neural networks of squid and octopuses to the complex
brain structures at the terminal end of the neural tube in vertebrates,
the variation across species is humbling**** **

**people may
claim that ��more advanced�� species like humans are the result of
an increasingly centralized nervous system that was produced through
evolution. ****This claim of advancement
through evolution is a common, but misleading, one****.
It suggests that evolution always moves in one direction: the advancement
of species by increasing complexity **

**evolution
may selectively enable body structures
that are more enhanced and complicated, ****but it may just
as easily enable species**** **

**that have
abandon complex adaptations
in favour of simplification.**** **

**Brains,
too, have evolved in the same way.
While the brains of some species, including humans,
developed to allow them to thrive, ****others have abandoned
their brains
because they are no longer necessary****.
**

**For example,
the ascidian, or sea squirt, lives in shallow coastal waters and which
is a staple food in certain regions, has a vertebrate-like neural structure
with a neural tube and notochord in its larval stage.**

**As the larvae
becomes an adult, however,
these features disappear until
only very basic ganglions remain. **

**In evolutionary
terms this animal is a ��winner��
because it develops a very simplified neural system better adapted to
a stationary life in seawater **

**In the long
run, however, evolutionary success will be determined by what species
survives longer: ****humans with their
complex brains ****(and
their weapons)****
or the brainless Dicyemida **

**1948-1990**

**���ɦ�έ�ææϦͦ�ς
�ӦϦ� ���ϦѦ̦�ά �ʦ��� ���ͦǦצ�ός �Ӧ�ς ���˦˦�ς ���˦Ŧ�ί�Ϧ�. **

**���Ŧͦ�ή�ȦǦʦ�
�ҦӦǦ� ����ή�ͦ�. **

**���Ŧ�ί�ͦǦҦ�
�ӦǦ� �ʦ��Ѧ�έ�Ѧ� �ӦϦ� �Ӧ� 1970 ����ό �Ӧ� ���ŦҦҦ��˦Ϧ�ί�ʦ�
�̦� �Ӧ� �ҦԦæʦ�ό�ӦǦ̦�-�ͦӦϦ�έ�Ӧ� "��ά�̦ئ� �ʦ���
���ɦͦ�ί��ς".**** **

**���� 1976
�ɦĦ�ύ�Ŧ� �Ӧ� �ҦԦæʦ�ό�ӦǦ̦� "���ЦԦѦɦĦ�ύ�˦�". **

**��
�Ҧ�έ�צ� �̦�ς
��ί�ͦ��� �Ӧ� ���զŦͦӦɦ�ό
ή
�� �ԦЦǦ�έ�Ӧ�ς �̦�ς ;**** **

*Emotional
Intelligence*** **

**also called ***EI*** or ***EQ ***,**** ****describes an ****ability****, ****capacity****, or ****skill****
to ****perceive****, ****assess****, and ****manage****
the ****emotions**** of one's self,
of others, and of groups**** **

**H ****�ЦϦɦǦӦɦ�ή �ͦϦǦ̦Ϧ�ύ�ͦ�****
�̦ЦϦѦ�ί �ͦ� �˦�ί�ЦŦ� ����ό �ӦϦ�ς �Ц��ͦӦϦæ�ώ�ҦӦ�ς, **

**�ʦ�
�ئҦ�ό�Ҧ� �ͦ� �ʦ��ӦϦɦʦ�ί
��έ�Ҧ� �ҦӦϦ� �Цɦ� ���Ц�ό�� ά�ͦȦѦئЦ� **

**Class-project
Oral-Exams**** **

**Oral-Exam
Appointments**** **

**Date**

**Time**

**1223**

** **

**1227**

** **

**
1023**

** **

**3**^{rd}**
hour**

** ****962**

**980**

**995**

**1202**

**923**

**950**

**979**

**1024**

**915**

**920**

**932 **

**949**** **

**2**^{nd }
**hour**

**627**

**887**

**946**

**960**

**711**

**809**

**874**

**909**

**794**

**845 **

**893 **

**899 **

**1**^{st}**
hour**

**7 June**

**5 June**

**31 May**

**AEM**

**Further Inquiries **** **

- Guide for Modeling and Calculating Shrinkage and Creep in Har~ened Concrete
- Canada: Financial Sector Assessment Program -- Crisis Management and Bank Resoltion Framework -- Technical Note; IMF Country Rep
- Untitled
- To whom it may concern:
- Rapportmall - English
- Hands-On Hardware Hacking and Reverse Engineering Techniques
- The eKOΝΕΣ Framework
- Chatter Publisher Actions and Salesforce1
- Untitled
- The End of Day process transmits your shipment information to UPS and prints the reports required by your UPS driver
- Dominant Ethnicity, Demography and Conflict: Why the devil is in the details
- FtLauderdale Overall Score Reports
- Plug-In Electric Vehicle Rebate Form
- Joint Language and Translation Modeling with Recurrent Neural Networks
- I. What is a Supermax prison?
- An Intrusion-Detection Model
- Manufacturing defects in the automobile industry, a case study of the remote causes and effects of Toyotas transmission malfunct
- High-Fidelity Terascale Simulations of Turbulent Combustion Jacqueline H. Chen, Evatt R. Hawkes, Ramanan Sankaran, James C. Sutherland, and
- QUESTION 1:
- Petri Nets: Properties, Analysis and Applications
- food project.nws

- Cholangiocarcinoma
- Lactate dehydrogenase A
- Aerobic glycolysis
- Video Tracking
- Target detection
- Camshift
- Kalman Filter
- FusionTracking
- Northern heating areas
- Existing buildings
- Energy Saving
- Fuzzy Clustering
- Suggestions
- Stock funds overall flow ratio
- Single fund flow ratio
- Stock market returns
- disabled
- Diseases of metaphor
- Religious spirit
- Heart and soul (xinhun)
- Thermally sensitive polymers

All Rights Reserved Powered by Free Document Search and Download

Copyright © 2011This site does not host pdf,doc,ppt,xls,rtf,txt files all document are the property of their respective owners. complaint#nuokui.com