We describe a system for inspection of formed tube using stereo vision. Model-based methods are used for robust and accurate location of tube edges. Tube centrelines are matched for stereo reconstruction by trinocular correspondence, and the complete 3D configuration of the tube confirmed by route following. Precision and repeatability values of the order of 0.1mm are obtained. There is no requirement for a prior CAD description of the tube, so that the system can be used both for process control and reverse engineering.

1 Introduction

1.1 Background

Formed metal tubes are an important component in a wide range of manufactured assemblies. Examples include exhaust and brake pipes in motor cars, heat exchangers and aircraft fuel lines. The three dimensional path of the bent tube is often quite complex, and high accuracy may be necessary in bending a tube to fit a prescribed conformation. Inspection of bent tubes may be required for several reasons: Quality control (ensuring that manufacture remains within tolerance) and associated process monitoring; Reverse engineering (developing a bending specification for a prototype geometry) or Precise specification of the manufacturing process.

The last requirement arises from the fact that, due to the properties of the materials, bending may result in systematic, but unpredictable, deviations from the intended geometry, resulting in the final product failing to conform to specification. If it is possible to measure accurately the difference between the intended conformation and the one that was actually manufactured, then the production parameters can be refined. The higher the accuracy and precision of the measurement, the fewer refinement cycles are required.

Users of inspection systems are tube-bending companies servicing a variety of manufacturing industries, principally automotive and aerospace. Measurement systems in current use fall into two categories. Contact measuring machines (CMM) provide highly accurate and precise measurements (to the level of microns), but are extremely slow. Tube-specific machines are in use which consist of a hand-guided 5-axis arm using infra-red non-contact V-probes. They are faster than CMM but less precise (tenths of a millimetre) and suffer the disadvantages that they are highly labour-intensive and require subjective definition of the measurement points. The objective of this project is to devise an automatic method of tube measurement which achieves accuracy and precision comparable with tube-specific devices. We have approached the problem using stereo vision.

While straightforward in principle, the difficulty of applying stereo vision techniques depends on the data source and the task. Application of stereo to industrial problems has mainly been concentrated on object recognition [1] or robot guidance [2,3]. This is generally more complex than extracting dense depth information from satellite images [4,5] or scanning electron micrographs [6], due to occlusion and absence of photogrammetric invariance in industrial environments. Despite technical success, these industrial applications have not made the last step into commercial application. This may be due to computational requirements and cost of systems. There also appears to be a perception (deserved or otherwise) of lack of algorithmic robustness for application of stereo outside the laboratory. Applications in three-dimensional metrology are also surprisingly rare. Such systems as have been reported are laboratory evaluation systems [7,8]. The system described here is the prototype of an instrument currently in production.

1.2 The Inspection Problem

Tubes are bent into complex configurations (see figure 1) in which occlusion is common from any viewpoint. Furthermore, the surface characteristics of the tubes may vary, depending on their application, from dark matt to highly reflective. These features result in the visual inspection problem being a challenging one.

The inspection requirement is to measure accurate lengths and relative orientations of the straight sections of a bent tube and to identify the connecting curved components. It should not be necessary to present the tube in any particular orientation for measurement, and the inspection method needs to be sufficiently robust to be used in an industrial environment.

Inspection takes place within an enclosed cell allowing lighting to be controlled and a degree of isolation from dust and vibration. The cell has an inspection volume of 1.5m ´ 0.75m ´ 0.5m with diffuse omni-directional illumination. Cameras are attached to a rack mounted on anti-vibration pads in such a way as to provide at least three non-collinear views of any part of the inspection area. The number of cameras required is determined by the complexity of the tube under inspection. For the complexity of tubes inspected to date 5 cameras have been used throughout the volume, each with a field of view of approximately 0.75m ´ 0.75m at the object.

We require the cameras to be calibrated for stereo measurement. We use the method due to Tsai [9] in which intrinsic and extrinsic parameters are estimated in separate stages. This method has been shown to deliver accuracy and precision of the order of 0.1mm over a distance of 1-2 metres (within specification for this application), and may be used with a planar calibration target under appropriate circumstances. This provides an advantage over intrinsically more accurate methods such as those arising from photogrammetry (e.g[10]). Algorithmic robustness, together with ease of manufacture and measurement and mechanical robustness of the target are important features in an industrial context [11].

2 Three-Dimensional Tube Measurement

We adopt the following measurement strategy.

1. Identify straight segments in each 2D view and extract the centreline . The tube centrelines, rather than the edges are the features to be matched for stereo reconstruction. As the individual stereo views may be widely separated, the tube edges visible in each view do not correspond in 3D, whereas the centrelines do. The centrelines are not physical features of the tubes, but are inferred from the 2D edge positions.
2. Reconstruct the centrelines of the straight segments in 3D . The camera arrangement in the cell is such that each section of tube is visible by at least three non-collinear cameras. This means that correspondences can be determined by trinocular coincidence, and the most appropriate pair of views can be used to determine the 3D orientation with greatest accuracy. This is a different strategy from that normally adopted in stereo, where individual features are matched. Using grouped features in 2D for matching adds robustness to the 3D result.
3. Determine the connectivity of the straight sections . Locating the centrelines and stereo matching is rather more difficult for curved sections than for straights, as curvature is not conserved under changes in viewpoint. The connectivity is determined by hypothesising 3D arcs joining straight segments and verifying these by tracking in the 2D views.
4. Locate the cut ends of the tube . Once the connections between straight segments have been identified, the tube can be described as a sequence of straight segments, whose orientations and extents are known with accuracy. It only remains to determine accurately the positions of the cut ends of the tube. This is achieved by fitting an ellipse to the image edge points around the end of the tube. The fitting process is constrained by the known orientation of the terminal straights.

In the following sections we give more details of the individual elements of this strategy.

2.1 Two-Dimensional Features

Tube edges usually display high contrast and are detected using the Canny operator[12]. Since the initial step is the location of straight sections, approximate edge positions (cues) are located by means of the Hough transform [13]. By setting thresholds on the Canny output, the size of Hough peaks and the size of allowable gaps between detected segments, the full extent of individual edge cues can be determined. Cues are further refined by matching opposing edges on the basis of parallelism, proximity and edge polarity.

These constraints, while generating a small, largely correct set of candidate straight sections, are not sufficiently robust to give completely reliable detection. False cues arise because of regions of low contrast, occlusions and surface reflections. Figure 2(a) shows the grey-level profile across an aluminium tube with a highly reflective surface. The contrast between the tube and the background is low in this case, and specular reflections give rise to strong edges within the tube boundary, which can confuse the edge-detection strategy. These difficulties are overcome using models of the expected distribution of grey-levels across the tube. Details of the application of this type of model are described by Woods et al . [14]. The models consist of statistical descriptions of the grey-level profile across a tube derived from training which capture variation in appearance of grey-level profiles across tubes arising from lighting variations etc. (figure 2).

The correct edge cues having been identified, the edge positions are determined by a gaussian fit to the gradient profile. In our hands, this allows edges to be located with a precision of about 0.1 pixel. El-Hakim and Pizzi [7] report a precision of about 0.05 pixel using a similar approach applied to highly ideal edges.

From these accurately placed pairs of edges, the 2D centrelines are determined by line fitting. Positions of the ends of the centrelines are determined using a threshold on the deviation of the edge positions from the fitted straight line.

2.2 3D Reconstruction

Initial reconstruction is of straight segments only. The correspondence problem becomes that of matching candidate centrelines from contributing 2D images.

There are at least three, sometimes more, views of each section of tube. For each pair of views, all possible line correspondences are used to generate candidate 3D lines. Correct matches are characterised by trinocular correspondence, which in this case is manifest as the coincidence (within a tolerance limit) of clusters of reconstructed lines corresponding to the correct matches from different image pairs. In practice, a cluster of two 3D lines generated from three pairs is sufficient to determine correspondence. (Partial occlusion may render one of the three possible members of the cluster unreliable). Not all pairs of stereo views will provide equal accuracy in the determination of the 3D line orientation. Generally speaking a better estimate will be obtained from a pair of views whose epipolar line is most closely perpendicular to the direction of the reconstructed line. The "best" 3D line is selected by projecting each member of a cluster back into the image (or images) not used in reconstructing its 3D orientation. The projection of an accurate 3D line will correspond closely to the observed 2D line in each of these images, and the measure for selecting the 3D line is the summed distance between the projected and observed 2D lines. Figure 3 shows the 3D lines generated from all possible pairings of 2D centrelines projected back into one of the 2D views (black lines). The white lines show the projections of those chosen as the correct reconstruction of the straight segments.

2.3 Determining Connectivity

To describe the conformation of the tube it is not only necessary to know accurately the orientations of the straight segments, but also their connectivity. Straights are connected by curved sections which are not detected by the 3D interpretation strategy, and so their presence and geometry must be inferred and verified. Tube bending machines insert bends by creating approximately circular arcs without twisting. Therefore a bend may be hypothesised between straights if they are coplanar. Proximity of the ends of a pair of straights is not a useful cue for proposing a bend between them, as long bends of low curvature are common. Each hypothesised bend is verified by proposing a connection in the form of a circular arc, projecting the arc into the 2D view which is most parallel to the plane of the bend, and seeking evidence for the tube at several positions along the arc (figure 4a).

2.4 Accurate End Positions

Once the connectivity is determined, the approximate positions of the two cut ends of the tube are known. To complete the measurement, these positions must be accurately determined.

In the current version of the system, we make the assumption that the tube is circular in cross-section and cut perpendicular to its length. The end will therefore appear on any image as an ellipse whose long axis is equal to the width of the tube and oriented perpendicular to the direction of the centreline. Depending on the viewpoint either half or the whole of the ellipse will be visible.

An approximate end position is found by searching along the centreline for image gradients oriented across the tube and fitting an ellipse to these positions (figure 4b). The tube end is further refined by searching for image gradients along radii of the fitted ellipse and repeating the fit to these positions. Our assumption of circular cross-section perpendicular to the tube means that the fit has only one parameter, namely the position along the 3D line. This constraint not only allows the fitting to be carried out efficiently, it also overcomes the stability problems for which ellipse fitting is notorious [15].

3 Performance

Performance of an inspection system is determined by accuracy and repeatability of measurements. These characteristics are determined in part by the computer vision methods employed and partly by the remainder of the system (mechanical stability, image resolution, frame-grabber precision etc.). The important parameters in determining how well a tube fits its specification are the positions of the ends of the straight segments. The measure typically used in quality control is the perpendicular distance of a measured endpoint from the vector representing the straight segment to which it belongs (figure 5). The sum of all these distances along the length of a tube gives a measure of how well the individual length and orientation measurements of straight segments have been made.

Using this measure we have tested the performance of the system in the following experiments.

Experiment 1 (Repeatability: electronic noise) : To test the underlying precision of the system, repeated measurements of the same tube were carried out without altering its position. This experiment gives us an idea of the underlying limits on performance due to imprecision in the image analysis methods and intrinsic variability in the images.

Experiment 2 (Repeatability: positional sensitivity and mechanical stability): Experiment 1 was repeated with the additional factor of moving the tube between measurements. The range of positions and orientations was chosen to cover the range of possibilities for the tube within the cell. Repeated measurements were carried out on each of 10 tubes of various lengths with diameters of 10, 25 and 60mm. This gives a measure of the precision and robustness of the measurements, and includes a component of mechanical stability as the door of the inspection cell is opened and closed between images.

Results of the experiments are shown in table 1. Each set of measurements on a given tube results in a set of geometries for the tube, from which an average geometry can be derived. The figures in table 1 are the mean and standard deviation of the projected distances of the end-points of straight sections from their corresponding centrelines in the average geometry (figure 5).

4 Discussion and Conclusions

The measurements in table 1 indicate that tube measurement is being made to a precision of about 0.1mm (mean displacement). The standard deviation of the measurements gives a measure of the robustness (repeatability) of the measurements. It is not possible from these figures to make any comment about accuracy, as this requires a "gold standard" against which measurements can be compared. One of the functions of the system is refinement of bending parameters. Measurement of the first tube of a batch is used to specify changes in the parameters for the remainder of the batch. The accuracy of the system is such that stable parameter settings are obtained after a single cycle.

Time of measurement in the current system depends on the complexity and size of tube, but is typically of the order of 25s. This is almost entirely processing time, and measured on our current configuration using a Pentium Pro 200MHz PC. The figure could be reduced immediately by using a faster processor, although there may be little advantage in doing so. The most time-critical application of this type of inspection is in manufacturing process monitoring, where cycle times of the order of 20s are typical

In addition to engineering a specific solution to the tube inspection problem, we have demonstrated the more general point that stereo vision is a potentially useful tool for accurate 3D inspection in an industrial environment. Precise and robust feature detection in 2D can form the basis of 3D measurement of sufficient precision and robustness for industrial metrology. Generally speaking, use of stereo vision requires some model of what is to be expected in order to make sense of the volume of information presented in the images. In the case of metrology, a CAD model can provide such an organising framework, and this has been used in some inspection projects. We use a more general model: that the object is composed of straight lengths of radially symmetric sections. This limits our system to the inspection of tubes, but allows a large proportion of inspection problems within the domain of tube inspection to be addressed. The model permits us to overcome a particular difficulty in this domain, namely that there are no identifiable features, such as corners, which correspond between views. Matching the centrelines and exploiting the linearity of straight segments has allowed us to build an efficient and robust analysis system.

The commercial importance of this application can be judged from the fact that two other groups have reported systems addressing this problem. Sinnreich and Bösemann [16] have described a similar approach to ours using stereo vision. They quote "accuracy of better than 0.5mm". Grimson et al . [17] take a different approach; they use a laser rangefinder to conduct a sequence of coarse and fine scans across the tube. They achieve high accuracy and repeatability (0.07 mm and 0.025 mm s.d. respectively, translated into the terms of table 1). Direct comparison with both these systems is difficult, as the values they measure may not correspond to those used here. It is not clear in either case, for example, what reference standard is used for accuracy. It would appear, however, that we are achieving a better inspection specification than [16] using a similarly engineered system. A rather higher inspection specification is achieved in [17] using more complex engineering in the form of a controllable robot arm. That approach is, however, constrained by the requirement for a CAD-like description of the tube geometry to guide the scanning path. This would make it inapplicable to some applications, such as reverse engineering.

The most thorough study of what is possible in metrology using stereo is that of El-Hakim and Pizzi [7], who have constructed and evaluated an experimental system. They measure the accuracy (bias) of their system to be 0.013 mm, evaluated using an idealised edge moved with high precision and verified by interferometry, and its precision to be 0.008 mm. They note that for best measurement, objects to be inspected should be kept away from the edges of the field of view, that images should be grabbed at least ten times and measurements made at least four times in different positions. The difference in measured precision between that system and the one described here is partly due (a factor of about 3) to their use of a much smaller measurement volume (30cm ´ 30cm ´ 20cm) and the fact that the recommended ideal measuring conditions have not been applied. We have also compromised towards robustness rather than maximum accuracy in calibration, both in terms of the planar target, and the algorithm used.

In operational terms, the precision and accuracy of the system is sufficient for a wide range of tube inspection applications. If higher accuracy and precision were required for a particular application, it could be achieved at the expense of additional cameras, optics and processing time. The results of [7] suggest that by doing so an improvement of a factor of two or three is the best that could be achieved.

References