04.03.2013
1
Einf��hrung in Visual Computing (EVC)
Image Processing & Computer Vision
186.822 VU 5.0 6 ECTS
Robert Sablatnig
1
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
EVC: Image Processing & Computer Vision
▪ Content:
▪ What are the
basic concepts of Image Processing and Computer
http://www.caa.tuwien.ac.at/cvl/teaching/sommersemester/evc
Vision and how are they used in
applications? The course answers
these questions by describing the
creation of
digital images using
digital cameras and the subsequent steps in order to derive
information kept in digital images automatically.
▪ A closer look is taken into
classical image processing techniques like
image
enhancement and
compression.
▪ The next step consists in the development of
digital filters and
segmentation techniques in order to be able to extract specific
information.
▪
Interest Points Computational Photography, 3D and motion are
further topics.
▪ Application of
Algebra and Analysis in reality
2
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
2
Logistics
▪ Lectures: 13:00 - 15:00
▪ Instructors: Robert Sablatnig (VO) and
g ( )
Sebastian Zambanini (UE)
▪ Textbook: 4 A4 pages available at Lectures and Website
▪ Further Reading:
▪ Richard Szeliski, Computer Vision: A Modern Approach
http://szeliski.org/Book/
▪ Sonka Hlavac Boyle: Image Processing Analysis and
Sonka, Hlavac, Boyle: Image Processing, Analysis, and
Machine Vision, 2nd Edition
▪ Webpage:
http://www.caa.tuwien.ac.at/cvl/teaching/sommersemester/evc
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
3
Readings
4
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
3
Introduction: What is Image Processing?
Computer Graphics vs. Computer Vision
6
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
4
Categorization
▪ Image Processing
▪
Manipulation of Image Data,
p
f
g
,
▪ Like removal of Noise, Correction of Sharpness on digital
images.
▪ Computer Vision
▪
Generation of non-graphical Data from images,
▪ Like Character-and Text Recognition, Segmentation of images
into „interesting�� parts, Detection of lines and corners.
„
g p
,
▪ Computer Graphics
▪
Generation of Images from non-graphical data,
▪ like bar charts, 3d graphics „VR�� in real time, graphical outputs
7
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Categorization
▪ Image Editing: Manipulation of Images (e.g. Photoshop)
▪ Visually
▪ Visually
▪ Interactive
▪ User-defined Parameters
▪ Image Processing: Mathematical algorithmic processes
▪ Image enhancement
g
▪ Image transformation (geometric)
▪ Image compression
▪ Image segmentation
8
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
5
Kategorisierung
9
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Example Image Processing: Filter (Noise
Removal)
10
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
6
Example Image Processing: Image
Enhancement
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
11
Example: Image Restoration
12
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
7
Example: Special Effects
13
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Difference: Pattern Recognition – Image
Processing?
▪ Pattern Recognition:
▪
Classification of
Patterns into a (finite)
number of pre-defined
(
)
p
classes
▪ like 2-dimensional patterns,
OCR
▪
Standard book: Duda and Hart 1973, "Pattern Classification and
Scene Analysis"
▪ Image Processing:
▪
Processing of an image to get a
new image that is
better suited
g
g
g
g
for a specific
task.
▪ Image
enhancement, image
transformation, image
compression, image
segmentation, image
restauration��
▪
Standard book: Rosenfeld and Kak 1982, "Digital Picture
Processing", 2nd Edition
14
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
8
Example Pattern Recognition
15
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Example Pattern Recognition
16
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
9
Examples for Pattern Recognition
17
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Example: Computer Vision
▪ Face Detection
18
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
10
Google Street View
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
19
Google Street View
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
20
04.03.2013
11
Introduction: What is Computer Vision?
Computer Vision
▪ Vision is derived from
Human
Vision (Human Visual System)
(
y
)
▪ Humans „see�� in
3 Dimensions
=> Computer Vision has 3d
components
▪ Evolution millions of years: Human
visual system
not faultless
∎ => if human visual system is not
faultless how can we expect
from a machine that it is?
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
22
04.03.2013
12
What is Computer Vision ?
"
Computer Vision describes the automatic deduction of the
structure and the properties of a (possible dynamic) three-
p p
f (p
y
)
dimensional world from either a single or multiple two-
dimensional images of the world"
- Vishvjit S. Nalwa:
A guided tour of computer vision. Addison-Wesley 1993
▪ Images: Color or Grayscale
C
Fi d
bl
▪ Camera: Fixed or movable
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
23
Computer Vision – Industry Related
▪ Computer Vision is an
exciting new
research area that studies how to make
computers efficiently
perceive, process,
and
understand visual data such as
images and videos. The ultimate goal is
for computers to emulate the
striking
perceptual capability of human eyes and
brains, or even to
surpass and assist the
human in certain ways. –
Microsoft
Research
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
24
04.03.2013
13
Computer Vision
▪ At least three goals:
1
Understand biological visual systems
1.
Understand biological visual systems
2. Build machines that see
3.
Understand fundamental processes of seeing
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
25
Computer Vision
We still do not know
▪ Is vision a well organized process with fundamental principles or
▪ a bag of tricks
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
26
04.03.2013
14
Goals and Applications of Computer Vision
▪ It is
not the goal of Computer Vision to develop a
robot that is
similar to humans [Whitney86]
[
y ]
▪ Goal is to
surpass and assist humans
▪ Applications:
▪ Automation (Assembly line)
▪ Inspection (Measuring of Parts)
▪ Remote Sensing (Maps)
▪ Human - Computer Interfaces
▪ Systems for Disabled
▪ Many more����
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
27
Computer Vision vs. Human Vision
▪ Why not
simply copy human vision researched by
neurophysiologists, psychologists, and psychophysics? [Levine91]
p y
g ,py
g ,
py p y
[
]
▪
Eye research is
finished –
Human Vision research is
not!
▪ Seeing is
not only a process within the eye – eye is only
producing images formed to ��impressions�� by the brain
▪ =>
Beginning of Computer Vision in the area of
Artificial
I llintegence
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
28
04.03.2013
15
Computer Vision vs. Seeing
▪ Seeing has adopted itself to environment und therefore not
faultless!
Is Seeing an integral part of intelligence?
▪ Is Seeing an integral part of intelligence?
▪ Do we see reality – or what we want to see?
▪
Is Seeing and Thinking separable?
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
29
It��s Just An Illusion: Visual Illusions
▪ Classical optical illusions
Zöllner Illusion (1860)
Poggendorf Illusion (1860)
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
30
04.03.2013
16
Visual Illusions
▪ Classical optical illusions
Helmholtz Squares (1866)
M��ller-Lyer Illusion (1860)
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
31
Visual Illusions
▪ Non existing 3D objects:
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
32
04.03.2013
17
Perspective Illusions by Julian Beever
Make Poverty History
Babyfood...
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
33
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
34
04.03.2013
18
M.C. Escher
Ambigious Interpretations
Indian vs. Inuit
Young/Old Lady
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
36
04.03.2013
19
Rotary Effects
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
37
Are these phenomena caused by manipulation of
the visual system by unreal images?
▪ After all, if we cannot believe what we see, what are we to
believe?
▪
Absolute faith in human visual system is
not justified for
2-
dimensional images!
∎ Either:
3d images -> real world
∎ or:
right limitations of scene features (perspective
∎ or:
right limitations of scene features (perspective,
lighting, direction, etc.)
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
38
04.03.2013
20
Are these phenomena caused by manipulation of
the visual system by unreal images?
Every image is an image of an object, which is understandable
only to those who know about its origins and are able to create a
y
g
corresponding image in their imagination (Helmholtz, 1910)
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
39
History
▪ Serious attempts to create computer vision systems have now a
history of 40-50 years.
y
y
▪ First digital image 1964 (Mariner 4)
▪ Focus on Sensors �� Digital Image Processing
▪ Analysis Focus ��
Computer Vision
Mariner 4: First
the first close-up
image ever taken
of Mars 1964
Source: NASA
Source: NASA
40
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
21
Computer Vision
The systems today are still exceedingly limited in their
performance ��
considerable room for improvement
p
p
41
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Future Challenges of Computer Vision
▪ Where do the innovations come from?
▪ 1. Hardware
Wavi Xtion
Kinect
▪ 2. Algorithms/Software
Kinect
http://www.gigapixel.com
42
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
22
Hardware
▪
First time that HW is no longer a real limitation !!
▪ Processing
▪ Image Resolution
▪ Storage
▪ Internet
▪ Mobile Devices
▪ Networks of cameras
43
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Processing
▪ Moore��s Law still holds!
▪ Multi-core CPUs
▪ Highly Parallel �� GPUs (+ Software eg. Cuda)
44
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
23
Processing
▪ What to expect:
▪ Image Processing (Feature Extraction) will be instantaneous
▪ Image Processing (Feature Extraction) will be instantaneous
▪ Real-time Libs: Basic algorithms (IPP,Cuda ��)
▪ Real time vision (cf. Real time Rendering)
▪ Parallelization (GPU implementation) is a feature of an algorithm
45
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Resolution
Ever growing resolution:
▪ 1975: 100 x 100 = 0 01 MP
▪ 1975: 100 x 100 = 0.01 MP
▪ 2009: 13.280 x 9.184 Pixel = 120 MP
▪ UltraCamx: 216 MP
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
46
04.03.2013
24
What can we do with that?
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
47
Some Questions to Tackle
▪ How/What shall we sample in space, time,
wavelengths, polarization, ��.?
g , p
,
48
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
25
Some Questions to Tackle
Optimal sampling strategies in 3D/4D ��
▪ No constraints of view-points
▪ No constraints of view points
▪ Multiple Images ��
Redundancy
▪ Control the illumination of each pixel
▪ How real are images?
49
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Storage
▪ We have huge disks and we fill them
▪ A color VGA image ~1 MB
▪ A color VGA image 1 MB
▪ Every 10 second 1 image
�� 8.4 GB a day
�� 240 TB a Life
▪ Can we index that? ��
Algorithms
50
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
26
Internet
Huge repository of images
▪ Flickr:
Aug 2011 ~ 6 Billion Photos On line
▪ Aug. 2011 ~ 6 Billion Photos On-line
▪ 4.5 million photo added per day
▪ YouTube:
▪ 65.000 new Videos a day
▪ 1 trillion video playbacks
▪ 20% of Internet Traffic
▪ Facebook:
▪ 600 Million users
▪ 3 Billion photos added per month
What can we do with these
images?
Source: Internet 2011 in numbers
51
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
What is Vision?
▪ What does it mean, to see? ��to know what is where by looking��.
▪ How to discover from images what is present in the world where
▪ How to discover from images what is present in the world, where
things are, what actions are taking place.
from Marr, 1982
52
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
27
The Importance of Images
▪ Some images are more important
than others
▪ 100 million $
��Dora Maar au Chat��
Pablo Picasso, 1941
53
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Where is now Computer Vision?
(only a few examples)
04.03.2013
28
Assisted Driving
Pedestrian and car detection
meters
meters
Ped
Ped
Car
Lane detection
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
55
• Collision warning systems
with adaptive cruise control,
• Lane departure warning
systems,
• Rear object detection
systems,
Iris Recognition
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
56
JOHN DAUGMAN
http://www.cl.cam.ac.uk/~jgd1000/iriscollage.jpg
04.03.2013
29
Image Stitching
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
57
Image Stitching
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
58
Brown, Lowe, 2007
04.03.2013
30
Photo Turism
PhotoSynth
Snavely et al. 2006
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
59
(Goesele et al. 2007).
Finding Paths through the World's Photos
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
60
04.03.2013
31
Why is Vision hard – The Plenoptic Function
The Structure of Ambient Light
62
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
32
The Structure of Ambient Light
63
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
The Plenoptic Function
Adelson & Bergen, 91
The intensity P can be parameterized as:
P (��, ϕ,
t,
��, Vx, Vy, Vz)
��The complete set of all convergence points constitutes the permanent possibilities of vision.��
Gibson
Image
coordinates
(sperical)
Color
Time
3D space
64
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
33
Measuring the Plenoptic Function
▪ ��
The significance of the plenoptic function is this: The
world is made of 3D objects but these objects do not
world is made of 3D objects, but these objects do not
communicate their properties directly to an observer.
Rather, the objects fill the space around them with the
pattern of light rays that constitutes the plenoptic
function, and the observer takes samples from this
function.�� Adelson & Bergen 91.
function. Adelson & Bergen 91.
65
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Measuring the Plenoptic Function
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
66
Why is there no picture appearing on the paper?
04.03.2013
34
Measuring the Plenoptic Function
▪ Light rays from many different parts of the scene strike the same
point on the paper.
p
p p
Forsyth & Ponce
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
67
Camera Obscura
68
04.03.2013
35
Measuring the Plenoptic Function
The camera obscura
The pinhole camera
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
69
Camera Obscura
▪ Latin:
▪ Camera for "vaulted
chamber/room"
▪ obscura for "dark"
▪ together "darkened
chamber/room��
"When images of illuminated objects penetrate through a small
▪ When images of illuminated objects ... penetrate through a small
hole into a very dark room ... you will see [on the opposite wall]
these objects in their proper form and color, reduced in size ... in a
reversed position, owing to the intersection of the rays". - Da Vinci
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
70
http://www.acmi.net.au/AIC/CAMERA_OBSCURA.html (Russell Naughton)
04.03.2013
36
Camera Obscura
▪ Used to observe eclipses (eg., Bacon, 1214-1294)
▪ By artists (eg Vermeer)
▪ By artists (eg., Vermeer).
71
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Camera Obscura
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
72
04.03.2013
37
Camera Obscura
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
73
Jetty at Margate England, 1898.
Pinhole Camera
▪ Simple Model of Camera Obscura: Pinhole camera
▪ Very small hole (aperture ~ 0), Light passes through the hole
and forms image on back (upside down and flipped)
74
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
38
Pinhole Camera
▪ Abstract camera model - box with a small hole in it
▪ Pinhole cameras work in practice
▪ Pinhole cameras work in practice
75
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Commercial Pinhole Cameras
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
76
04.03.2013
39
Playing with Pinholes
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
77
Effect of Pinhole Size
Wandell, Foundations of Vision, Sinauer, 1995
78
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
04.03.2013
40
Limits of Pinhole Cameras
▪ A picture of a filament taken with a pinhole camera. In the image
on the left, the hole was too big (blurring), and in the image on
,
g (
g),
g
the right, the hole was too small (diffraction).
Ruechardt, 1958
79
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
Pinhole Camera Images with Variable
Aperture
▪ Why not making the aperture
as small as possible?
l h
h
h
2 mm
1 mm
0.6 mm
0.35 mm
▪ Less light gets through
▪ Diffraction effect
80
Robert Sablatnig, Computer Vision Lab, EVC-2: Introduction
0.15 mm
0.07 mm