Image of the inside of the Bike Hut in San Gregorio
with a window seen at an angle
This project involved finding correspondences between two images, computing a homography matrix for the two images and their correspondences, and transforming one of the images using the homography matrix in order to form a mosaic.
I had a lot of fun working on the project, and learned a lot about homography matrices and how panoramic images work.
To start, I took three sets of images that I intended to warp into mosaics. Two of the image sets were taken specifically for the purposes of this project, and I took the other while traveling last summer.
I also found two images with noticeable shapes to rectify, both of which I took a few months before this project.
Then, I used the correspondence tool given to us in Project 3 to find correspondences between the images for the mosaic, and find the relevant points for the rectification images.
Specifically, my first two mosaics contained 3 images each. For these mosaics, I used the tool to find both the correspondences between the first image and the second image, and the second image and the third image. For the images I hoped to rectify, both contained windows taken at an angle, and I aimed to transform these images so that the windows would be transformed to a square shape for both images.
After finding these correspondence pairs for the mosaic images, I wrote a function to compute the homography matrix, H, given the correspondence pairs.
Specifically, I did this by manipulating the equation p' = Hp - transforming the first eight elements of H (all but the bottom right element, which is always 1) to a length-8 vector that we're trying to solve for. By manipulating the expression for H * p, we can add a matrix M that contains two entries for each correspondence point, and eight columns. We also create a vector b with two entries for each correspondence point, one entry each for the x-coordinate and y-coordinate, respectively. For every correspondence point pair, we can calculate the appropriate elements to put in its two corresponding rows, and we can solve for the equation Mh = b using least-squares, finding an optimal value of h, from which we can reconstruct the H matrix.
After computing the H matrix, I made a function to warp one image to another given the H matrix of the correspondences between the two images, by first using skimage.draw.polygon to get all the points within the bounding box of an image, taking the dot product of H with all these points, and using scipy.interpolate.griddata to interpolate the pixel values from the original image on the new transformed polygon.
Before warping the images into mosaics, I used the warp function to rectify two images. The first image was taken from the inside of the Bike Hut in San Gregorio, with a window at a diagonal angle. The other was an image of the outside of a boba shop taken at a diagonal angle. For both of these images, I used the correspondence tool to find the points for each of the four corners of a window in the image, and computed the H matrix for this image and four points that formed a square. Then, I applied my warp function to each image with these H matrices, resulting in the window in each image being transformed into a square shape.
After having created this homography matrix computation function and warp function, I applied it to the sets of images I intended to create mosaics of. For the sets of three images, I first warped the first image to the second, and warped the third image to the second, and combined all of these images. For the two-image mosaic, I simply warped the first image to the second, and combined them.
For all of them, I combined them by modifying the alpha values of each image: for each image, before transforming it, I made the alpha values at the horizontal center of the image 1, made the alpha values 0 at the left and right edges, and had the alpha values increase linearly from 0 to 1 towards the center. From here, I took a weighted average of the pixel values for the three images after transformation, with the weights corresponding to the alpha values computed earlier. After averaging this for all pixels, I ended up with a panoramic image mosaic! I dealt with out-of-bounds issues by estimating the width and height of the final mosaic image, and both having the final image initialized with this size, and centered at (0, 0) in the original middle image. This way, even if some of the transformed images had negative position values, they would still be shown in the final image.
I applied these steps to three sets of images: three pictures I took inside the VLSB library, three pictures I took from a forest on campus, and two pictures I took from the top of Mt. Constitution on Orcas Island in Washington.
Once I had working code for creating image mosaics with manual correspondence lists, I proceeded to add new code to create image panoramas automatically, by computationally finding correspondence points.
First, I used the Harris corner detection to find possible "corner points" in the image. I did this using the example Python code provided in the project description.
After this, I applied the techniques from Section 3 of the paper given to us in the project description to use MOPS to filter out only the top N corner points with sufficient distance from others: specifically, I chose N to be 100 for this project, and sorted points by r_i distance, choosing the top 100 points after sorting. A point's r_i value is calculated as the minimum distance the point is away from another point with a higher Harris Corner score after multiplying by a factor of 0.9 (to account for points that are nearby and have very close Harris Corner values)./p>
From here, I had a list of candidate correspondence points in each image, and I generated an 8 by 8 image patch for each corner, by using skimage.transformation.resample to generate an image with 20% of the size of the original image, and use this to have an 8 by 8 patch in the resampled image correspond to a 40 by 40 patch in the original image.
Once I had these image patches, I took each patch in Image 1, and found its nearest neighbor in Image 2. To find as many inliers as possible, I used Lowe's Trick to filter points based on the ratio of the match quality with the first nearest neighbor to the match quality with the second nearest neighbor, choosing a parameter of 0.75 for this project (e.g. only choosing points with a nearest-neighbor ratio of less than 0.75).
After this, I ran RANSAC 1000 times to find a set of correspondence points containing only inliers. Once I had these points, I ran the correspondence-point-finding algorithm on all of my image pairs from Part 1, and produced image mosaics in the same way using my code from Part 1.
In the end, the automatic point finding worked as expected on the library and forest images, and it did not work on the Orcas Island image. This is likely because the Orcas Island image contains less overlap than is recommended (around 20% overlap) and a large portion of the image is either of ocean or blue sky, which are hard to find correspondence points in.
To compensate for this, I took one more mosaic images, which is shown here in addition to the ones from Part 1 that the algorithm worked on.
Overall, I had a lot of fun working on the project, and learned a lot about image stitching and image mosaics. Learning-wise, the most interesting thing I took away from this project was in the power of the RANSAC algorithm - before learning it in lecture, it didn't seem like it'd be possible to filter out all the outliers from the set of possible correspondences. But with a few fairly simple lines of code implementing RANSAC, it was cool to see that the algorithm magically worked and always selected correct correspondence points whenever possible!