A common method for construction of panoramic mosaics includes the collection of sections (``strips'') from each image and pasting these strips next to each other to become the mosaic. In the simple case of a camera which is moving horizontally, vertical strips are taken from each image and pasted side by side [ 11 ] (see Fig. 1 .a). This process can also be viewed as scanning the scene with a vertical broom [ 4 , 16 ]. This vertical broom scans the entire sequence, extracts vertical strips along the sequence, and pastes them one next to the other to create the panoramic mosaic. The vertical broom is perpendicular to the horizontal optical flow, and after placing the strips in the panoramic image, the optical flow is pointing exactly to the direction from which the panoramic image is constructed (see Fig. 1 .b).
Figure 1:
The relation between the mosaicing process using strips and the
direction of the optical flow.
(a) The simple case of camera which is moving to the left. The optical
flow points to the right, and vertical strips are collected. After
pasting, The optical flow is parallel with the direction in which the
panoramic image is built.
(b) New information is passing through the scanning broom when the
Optical Flow is perpendicular to the broom.
(c) No new information is passing through the scanning broom when it is
parallel to the optical flow.
(d) When the scanning broom is set to be perpendicular to a radial
optical flow its shape will be a circular arc.
The scanning broom must be perpendicular to the optical flow. A counter example is shown in Fig. 1 .c, using a vertical scanning broom with vertical optical flow. When the optical flow is parallel to the scanning broom no mosaic is created as no new information will pass through the broom. The case where the scanning broom is perpendicular to a radial optical flow is shown in Fig. 1 .d. The information from all images in the sequences will pass through the scanning broom, allowing to collect strips for pasting in the mosaic.
The following section describes a practical implementation for determining the shape of the scanning broom for the case of affine motion, which can give good approximation for image motion in many cases. Numerous methods exist to recover the parameters of an affine transformation [ 7 , 13 ]. Implementations for other motion models can be done in a similar manner.
The affine transformation can be expressed as:
where
and
are corresponding points in images
and
, and the parameters of the affine transformation
are (
a
,
b
,
c
,
d
,
e
,
f
). (
u
,
v
) is the optical flow vector as a function of the position
. The transformation
(and the optical flow) vary continuously along the sequence.
The scanning broom will be a line
such that it will be perpendicular to the optical flow. The normal to
the line
is in the direction
thus it should be in the same direction as (
u
,
v
). This constraint can be expressed by:
for some value of
k
. A line equation can be derived by integration only when
e
=
c
. In most cases, the difference between the values of
c
and
e
is due to the rotation around the optical axis
, such that it contributes
to
c
, and
to
e
. To approximately satisfy the condition
e
=
c
it is therefore sufficient to rotate the image by
after the affine transformation is recovered, and then recompute the
affine transformation.
After correction for rotation around the optical axis, when the affine
transformation is such that
, we get the following equation of the scanning broom by integration:
This is a family of lines that are all perpendicular to the optical flow. M is used to select a specific line. We suggest that M will be set to the value for which the line contains maximum number of pixels within the image. If many options exist, then we suggest using a line as close as possible to the center of the image to minimize lens distortions. This selection should ensure that pixels used in the mosaic will be from that image having best resolution at that location.
We will use the following notation to describe the scanning broom along
the sequence: The line
is the line in Image
, in it's coordinate system, which corresponds to the affine
transformation
. This affine transformation
relates points
in Image
to corresponding points
in Image
(see Fig.
3
).
Eq. 3 can be easily understood for some simple cases.
takes the form
, thus the selected broom becomes
, which is a vertical line (see Fig.
2
.a).
, thus the selected line becomes
, which is a horizontal line (see Fig.
2
.b).
, where
b
is the scaling factor (
f
=
b
). As a result, the selected broom becomes
, which is a circle around the center of the image (see Fig.
2
.c).
For pure camera translations the scanning broom will be a circle around the Focus of Expansion (FOE). In more general cases the scanning broom may be approximated by an elliptic curve.
Figure 2:
Examples for scanning broom.
(a) A vertical scanning broom is selected for horizontal motion.
(b) A horizontal scanning broom is selected for vertical motion.
(c) A circular scanning broom is selected for zoom and for forward
motion.
The mosaic is constructed by pasting together strips taken from the original images. The shape of the strip, and its width, depend on the image motion. This section describes how to determine these strips in the case of an affine motion to conform with the methodology of the selection of best resolution. Strip selection for other motion models can be done in a similar manner.
In order to determine the strip to be taken from Image
, the preceding frame,
, and the succeeding frame,
, should be considered. Let
be the affine transformation relating points
in Image
to the corresponding points
in Image
, and let
be the affine transformation relating points
in Image
to the corresponding points
in Image
.
Given the affine transformations
and
, the lines
and
are selected respectively (see Fig.
3
.a-c). The line
in
corresponds to the line
in
using the affine transformation
. In the same way, the line
in
corresponds to the line
in
using the affine transformation
.
The strip that is taken from the image
is bounded between the two lines
and
in
(see Fig.
3
.a-c).
Using this selection, the first boundary of the strip will be described
by the selected line
, thus will be exactly orthogonal to the optical flow with regard to the
previous image. The second boundary of the strip is described by the
line
which is the projection of the line
onto the current image
, having the same property in the next image.
This selection of the boundaries of the strip ensures that no information is missed nor duplicated along the strip collection, as the orthogonality to the optical flow is kept.
Figure 3:
Example of cutting and pasting strips.
(a)-(c) Strips are perpendicular to the optical flow. Line
is selected in Image
and Line
is selected in Image
. The mapping of Line
(in
) into Image
using the affine transformation is Line
. The strip
taken from Image
is bound between lines
and
.
(d) Strips are warped and pasted so that the optical flow becomes
parallel, their back is fixed (e.g.
in Strip
) and their front (e.g.
in Strip
) is warped to match the back of the next strip.
To create continuous mosaic images while avoiding accumulated distortions, the warping of the strips should depend only on the adjacent original frames, independent of the history of previous distortions. This section describes how to warp the strips in the case of affine motion.
In our scheme, the back of each strip is never changed. This is the side
of the strip which corresponds to the boundary between Image
and Image
and defined by
. The front of the strip is warped to match the back side of the next
strip. This is the boundary between Image
and Image
which is defined by
.
In the example described in Fig. 3 .d, we warp the first strip such that its left side does not change, while its right side is warped to match the left side of the original second strip. In the second strip, the left side does not change, while the right side is warped to match the left side of the third strip, etc.
As a result, the constructed image is continuous. Also, if we warp the original optical flow as we did with the strips, the resulting flow will be parallel to the direction in which the panoramic mosaic is constructed. Moreover, no accumulative distortions are encountered, as each strip is warped to match just another original strips, avoiding accumulative warps.
The strips as pasted in the example of Fig. 3 remain curved. The transformation can be modelled by a projection into a cylindrical manifold, in which case the strips should be un-bent into straight edges before pasting into the mosaic. Un-bending the strips may cause distortions in some parts of the mosaic, and before visual presentation the mosaic should be re-bent depending on the desired viewpoint. The bending and un-bending, however, is not very critical when the strips are curved only slightly.
Adrian F Clark
