Optimizing and Learning for Super-resolution

 
 

Optimizing and Learning for 
Super-resolution 

Lyndsey C. Pickup, Stephen J. Roberts

& Andrew Zisserman

Robotics Research Group, University of Oxford

 
 

The Super-resolution Problem 

Given a number of low-resolution images

differing in:

• geometric transformations
• lighting (photometric) transformations
• camera blur (point-spread function)
• image quantization and noise.

 

Estimate a high-resolution image:

 
 

Low-resolution image 1

 
 

Low-resolution image 2

 
 

Low-resolution image 3

 
 

Low-resolution image 4

 
 

Low-resolution image 5

 
 

Low-resolution image 6

 
 

Low-resolution image 7

 
 

Low-resolution image 8

 
 

Low-resolution image 9

 
 

Low-resolution image 10

 
 

Super-Resolution Image

 
 

Generative Model 

Registrations, lighting and blur. 

High-resolution image, x. 

y1 

y2 

y3 

y4 

Low-resolution images 

W4 

W3 

W2 

W1

 
 

Generative Model 

• Geometric registrations
• Point-spread function
• Photometric registrations

 

We don’t have: 

We have: 

• Set of low-resolution input images, y.

 
 

Maximum a Posteriori (MAP) Solution 

Standard method: 

1. Compute registrations from low-res images.
2. Solve for SR image, x, using gradient descent.

 

y1 

y2 

y3 

y4 

W4 

W3 

W2 

W1 

x 

[Irani & Peleg ‘90, Capel ’01, Baker & Kanade ’02, Borman ‘04]

 
 

What’s new 
 

1. We solve for registrations and SR image jointly.

1. We also find appropriate values for  
    
    
    
   

  parameters in the prior term at the

  same time. 

• Hardie et al. ’97: adjust registration while optimizing super-resolution estimate.
• Exhaustive search         limits them to translation only.
• Simple smoothness prior         softens image edges.

 
 

i.e. given the low-res images, y, we solve for the SR image x and the mappings, W simultaneously. 

y1 

y2 

y3 

y4 

W4 

W3 

W2 

W1 

x

 
 

Overview of rest of talk 
 

• Simultaneous Approach
• Model details
• Initialisation technique
• Optimization loop

• Learning values for the prior’s parameters 
  

• Results on real images 
  

 
 

Maximum a Posteriori (MAP) Solution 

Image x. 

Corrupt with additive Gaussian noise. 

Warp, with parameters Φ. 

Blur by point-spread function. 

Decimate by zoom factor. 

y1 

y2 

y3 

y4 

W4 

W3 

W2 

W1 

x 

y

 
 

Details of Huber Prior 

Huber function is quadratic in the middle, and linear in the tails. 

Probability distribution is like a heavy-tailed Gaussian. 

ρ (z,α) 

p (z|α,v) 

Red: large α

Blue: small α 

This is applied to image gradients in the SR image estimate.

 
 

Details of Huber Prior 

Ground Truth 

α=0.1 v=0.4 

Too little smoothing 

Too much smoothing 

α=0.05 v=0.05 

α=0.01 v=0.01 

α=0.01 v=0.005 

Edges are sharper 

Advantages: simple, edge-preserving, leads to convex form for MAP equations.  

Solutions as α and v vary:

 
 

Advantages of Simultaneous Approach 

• Learn from lessons of Bundle Adjustment: improve results by optimizing the scene estimate and the registration together.

• Registration can be guided by the super-resolution model, not by errors measured between warped, noisy low-resolution images. 
  

• Use a non-Gaussian prior which helps to preserve edges in the super-resolution image. 
  

 

 
 

Overview of Simultaneous Approach 

1. Start from a feature-based RANSAC-like registration between low-res frames.

1. Select blur kernel, then use average image method to initialise registrations and SR image. 
   

1. Iterative loop: 
   

 

• Update Prior Values
• Update SR estimate
• Update registration estimate

 

 
 

• Use average image as an estimate of the super-resolution image (see paper).

• Minimize the error between the average image and the low-resolution image set. 
  

• Use an early-stopped iterative ML estimate of the SR image to sharpen up this initial estimate. 
  

 

Initialisation 

Average image 

ML-sharpened estimate

 
 

1. Update prior’s parameter values (next section)

1. Update estimate of SR image 
   

1. Update estimate of registration and lighting values, which parameterize W 
    
    
   

• Repeat till converged. 
   
   
  

 

Optimization Loop

 
 

Joint MAP Results 

Decreasing prior strength 

Registration Fixed 

Joint MAP

 
 

Learning Prior Parameters α, ν  

• Split the low-res images into two sets:

 

Use first set to obtain an SR image. 

Find error on validation set.

 
 

Learning Prior Parameters α, ν  

• Split the low-res images into two sets:

 

Use first set to obtain an SR image. 

Find error on validation set. 

• But what if one of the validation images is mis-registered?

 
 

Learning Prior Parameters α, ν  

• Instead, we select pixels from across all images, choosing differently at each iteration.

 

• We evaluate an SR estimate using the unmarked pixels, then use the forward model to compare the estimate to the red pixels.

 
 

Learning Prior Parameters α, ν  

• Instead, we select pixels from across all images, choosing differently at each iteration.

 

• We evaluate an SR estimate using the unmarked pixels, then use the forward model to compare the estimate to the red pixels.

 
 

Learning Prior Parameters α, ν  

• To update the prior parameters:

 

2. Re-select a cross-validation pixel set.
2. Run the super-resolution image MAP solver for a small number of iterations, starting from the current SR estimate.
3. Predict the low-resolution pixels of the validation set, and measure error.
4. Use gradient descent to minimise the error with respect to the prior parameters.

 

 
 

Results: Eye Chart 

MAP version: fixing registrations then super-resolving 

Joint MAP version with adaptation of prior’s parameter values

 
 

Results: Groundhog Day

 
 

• The blur estimate can still be altered to change the SR result. More ringing and artefacts appear in the regular MAP version.

 

Results: Groundhog Day 

Blur radius = 1 

Blur radius = 1.4 

Blur radius = 1.8 

Regular MAP 

Simultaneous

 
 

Lola Rennt

 
 

Real Data: Lola Rentt

 
 

Real Data: Lola Rentt

 
 

Real Data: Lola Rentt

 
 

Real Data: Lola Rentt

 
 

Conclusions 

• Joint MAP solution: better results by optimizing SR image and registration parameters simultaneously.

• Learning prior values: preserve image edges without having to estimate image statistics in advance. 
  

• DVDs: Automatically zoom in on regions with a registrations up to a projective transform and with an affine lighting model. 
  

• Further work: marginalize over the registration – see NIPS 2006.