The most common approaches to generating dense optical flow fields use the intensity constancy constraint but employ different sources of additional information to constrain fully the 2D visual motion at each pixel. These include regularisation using local smoothness constraints[ 3 ]; the multi-constraint approach employing additional constancies such as intensity gradient[ 10 ]; and neighbourhood-based parametric motion models [ 1 ]. In common with many other examples of the parametric type, the proposed algorithm uses motion models[ 2 , 4 , 6 , 9 ] but differs both in the formulation of the motion estimator and in the process architecture of the algorithm. This architecture can make a significant impact on the ability to address the following important issues: coping with multiple motions , capturing large scale motions and segmenting consistent moving regions.
Typically methods based on motion models are implemented within a multi-resolution framework using a Laplacian pyramid [ 1 , 9 , 4 ]. Such an approach has the potential ability to capture large motions reducing the likelihood of aliasing. Coarse motion estimates can be rapidly generated at lower resolutions, and used to seed more accurate estimation as well as guide segmentation at finer resolutions. In practice, relatively few layers are employed because of the considerable blurring of object structure and merging of regions containing multiple motion at lower resolutions in the hierarchy. A multi-pass pixel segmentation strategy is usually embedded within this framework using robust statistical estimators[ 9 , 4 ]. The motion parameters of the largest (remaining) motion regions are propagated and refined down through the hierarchy while moving regions are identified as inliers at the finest level.
The above framework treats the image sequence as a series of independent processing problems where motion results from one frame do not naturally guide the analysis of subsequent frames. Moreover, the resultant motion parameters do not contain any explicit temporal components and may only be interpolated between frames or extrapolated beyond frames by assuming the pixel velocities remain constant. The problem becomes more apparent given the two application areas which have motivated our development of the optical flow method: frame interpolation for cinematographic special effects and video compression. Figure 1 (a) shows the position of an image point in four key frames. When altering the film rate of an image sequence, or predicting frames in motion compensated video compression, the true intermediate positions of an image point are required as shown in figure 1 (b). The lack of temporal components in the motion model, however, restricts interpolation to the linear form depicted in figure 1 (c).
This paper presents a new generalised formulation of the motion-model based optical flow estimator. This formulation, described in section 2 , has the following advantages. First, given an initial estimate of the motion field, the method can cope with arbitrarily large motions. Second, motion fields generated by other techniques such as block matching can be incorporated into the estimation process. Finally, spatio-temporal motion models which make explicit the temporal behaviour of the motion field can be generated. Examples of these behaviours and their relationship to the cinematographic and compression applications are presented in sections 2 and 5 .
Graeme Jones