From Homography to Image Stitching

  • 54 浏览

cold

2020/08/19 发布于 技术 分类

This presentation was on 2020.8.3, introducing the topic of image stitching. First, I clarify the deduction process and application constraints of homography, which I found was the most important concept in terms of image stitching. After that, I gave the demo of the general pipeline of image stitching, followed by detailed introduction on several classic works, including AutoStitch, DWH, SVA, APAP, AANAP, Seam-driven methods.

文字内容
1. From Homography to Image Stitching 张寒萌 2020.08.03 1
2. Homography 2
3. Homography y x x y z z H1 z H2 y x • In camera calibration, we take photos of the checkboard from different perspectives. • For each image, we detect the inner corners and thus calculate a homography between the image and the checkboard plane. 3
4. Homography • From world coordinate to camera 1 coordinate, 1 1 • Suppose the checkboard plane falls on z = 0, we have 1 1 1 • Let , we derive 1 1 Case1: homography from a realworld plane to an image 4
5. Homography • Perform the same conduction for both H1 and H2, we have 1 1 2 2 1 2 • From image1 to image2: -1 Case2: homography from an image to another image 5
6. • For case1, from image1 to plane: • Using calibration board, we can naturally set (X, Y, Z) of keypoints in the world coordinates, thus calculating homography. 1 1 1 • For case2, from image1 to image2: • By establishing point-correspondences, we can derive their homography. 6
7. Homography • Homography can describe the transformation • between an image and a plane • between two images take from the same scene, but only under the condition that the scene is planar!!! No parallax(no image depth) The scene is roughly planar (or far enough) or Views differ purely by rotation (no translation) 7
8. parallax • Parallax problem in image stitching. • Notice the horizontal spatial order of the tower and the red sculpture. 8
9. Image stitching 9
10. pipeline feature extraction feature matching calculate H using RANSAC Image warping Image blending • Let us compose these images into a larger image 10
11. pipeline feature extraction feature matching calculate H using RANSAC Image warping Image blending • Find keypoints and represent by descriptors • Most classic descriptors: SIFT, SURF, ORB… 11
12. pipeline feature extraction feature matching calculate H using RANSAC Image warping Image blending • Matching features and establish point-correspondences. • Matching methods: Brute-Force Matcher, crossCheck matcher, KNN matcher, FLANN matcher 12
13. pipeline feature extraction feature matching calculate H using RANSAC Image warping Image blending • RANSAC: Random Sample Consensus • Remove wrong correspondences and calculate H 13
14. pipeline feature extraction feature matching calculate H using RANSAC Image warping Image blending • Represent the images in common coordinates with translation • Now we can compose the images 14
15. pipeline feature extraction feature matching calculate H using RANSAC Image warping Image blending
16. challenges Repeated texture weak texture foreground occlusion Large parallax 16
17. Autostitch • Autostitch • An early and classic method for image stitching, widely used in many applications like OpenCV stitcher, Photoshop… • algorithms • K-d tree • Bundle adjustment • Multi-band blending Brown M, Lowe D G. Recognising Panoramas [C]. ICCV. 2003 Brown M, Lowe D G. Automatic Panoramic Image Stitching using Invariant Features [J]. IJCV. 2007. 17
18. DWH • Divide into all feature points into two cluster using K-means. For each cluster, calculate its Homography and thus get and • The relationship between two overlapping images uses a blending of two homographes, Hg and Hd, expressed as: (1) • Assign a weight to each pixel location pij based on spatial proximity to determine how much of each homography should be used. (2) where and are the distances to the closest feature points in the sets and respectively Gao J, Kim S J, Brown M S. Constructing image panoramas using dual-homography warping [C]// CVPR, 2011. 18
19. SVA • Model the relationship of two images with a set of Affine transformations. ps: Affine transformation has 6 degrees of freedom, while projective transformation has 8 degrees of freedom Lin W Y, Liu S, Matsushita Y, et al. Smoothly varying affine stitching [C]// CVPR, 2011. 19
20. APAP • APAP: • As-projective-as-possible • objective: • find warps that aim to be globally projective, yet allow local deviations to account for model inadequacy Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible image stitching with moving DLT [C]// CVPR, 2013. 20 Zaragoza J, Chin T J, Tran Q H, et al. As-Projective-As-Possible Image Stitching with Moving DLT [J]. TPAMI, 2014.
21. APAP • DLT(Direct Linear Transformation) • A basic method to estimate H from a set of point matches across and • Let and be matching points across overlapping images, A projective warp or homography aims to map to following the relation (1) • Eq.(1) is rewritten as the implicit condition and linearised (2) 2 • Only two of the rows are linearly independent,Let i-th . DLT estimates the nine elements of H as be the first-two rows of (2) computed for the (3) • The solution is simply the least significant right singular vector of A. Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible image stitching with moving DLT [C]// CVPR, 2013. 21 Zaragoza J, Chin T J, Tran Q H, et al. As-Projective-As-Possible Image Stitching with Moving DLT [J]. TPAMI, 2014.
22. APAP • Moving DLT(Moving Direct Linear Transformation) • Main idea is to warp each using a location dependent homography (4) • where H∗ is estimated from the weighted problem (5) • The scalar weights change according to x∗ and are calculated as (6) Here, σ is a scale parameter, and xi is the coordinate in the source image I of one-half of the i-th point match • The problem in (5) can be written in the matrix form (7) • This is a weighted SVD problem, and the solution is simply the least significant right singular vector of Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible image stitching with moving DLT [C]// CVPR, 2013. 22 Zaragoza J, Chin T J, Tran Q H, et al. As-Projective-As-Possible Image Stitching with Moving DLT [J]. TPAMI, 2014. .
23. APAP • Efficient Learning for Image Stitching • Partitioning into cells • partition the source image I into a grid of C1×C2 cells. • For each cell, the centre coordinate is chosen as x∗ , and all pixels within the same cell are warped using the same H∗ . Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible image stitching with moving DLT [C]// CVPR, 2013. 23 Zaragoza J, Chin T J, Tran Q H, et al. As-Projective-As-Possible Image Stitching with Moving DLT [J]. TPAMI, 2014.
24. AANAP Based on APAP, (1) linearize the homography in the regions that do not overlap (2) estimate a global similarity transform using a subset of corresponding points in the overlapping regions (3) interpolate smoothly between the homography and the global similarity in the overlap- ping regions Lin C C, Pankanti S U, Ramamurthy K N, et al. Adaptive as-natural-as-possible image stitching [C]// CVPR, 2015. 24
25. Seam-Driven • • Mainstream: • step1 estimate H with the best geometric fit to align the images, using RANSAC; • step2 post-process, like seam-cutting Their work: • seam-driven image stitching strategy: evaluate the goodness of a transform based on the resulting visual quality of the seam-cut Gao J, Li Y, Chin T J, et al. Seam-Driven Image Stitching [C]// Eurographics (Short Papers), 2013. 25
26. Seam-Driven • • Generating homography candidates • First, standard RANSAC is applied with a comparatively large n (500 in implementation) to fit the dominant plan • After this, remove the inliers of the dominant plane and apply a sequence of RANSAC instances each with a smaller n (50 in implementation) on the remaining matched points. • This helps to find homographies that align other smaller planar structures which may result in better seam-cuts Computing the Seam-Cut • The seam computation can be formulated as a labeling problem on a Markov Random Field (MRF) which minimizes a global energy with the following form: where the data-cost of each pixel is defined to be the gradient at that location: the smoothness cost between two pixels p and q is defined as:'>as: • Evaluating the Cut • For each pixel, p, along the seam, we estimate an error value, E(p) by extracting a 17 × 17 patch, P, centered at p and searching for its most similar patch in either I1 or I2. This can be expressed as:'>as: Gao J, Li Y, Chin T J, et al. Seam-Driven Image Stitching [C]// Eurographics (Short Papers), 2013. 26
27. Seam-Driven • Evaluating the Cut • For each pixel, p, along the seam, we estimate an error value, E(p) by extracting a 17 × 17 patch, P, centered a and searching for its most similar patch in either I1 or I2. This can be expressed as: Gao J, Li Y, Chin T J, et al. Seam-Driven Image Stitching [C]// Eurographics (Short Papers), 2013. 27
28. Seam-Driven: RANSAC Intro • RANSAC: Random Sample Consensus • RANSAC loop: • Select four feature pairs at random • Compute homography H (exact) • compute inliers where SSD(p, H(p)) < ε • Keep largest set of inliers • Re-compute least-squares H estimate on all of the inliers • This random sampling process is repeated n times and the hypothesis with the highest consensus size is chosen. • Gao J, Li Y, Chin T J, et al. Seam-Driven Image Stitching [C]// Eurographics (Short Papers), 2013. 28