3 Assessment

In order to assess the performance of the system, a manual region-classification was performed on the most significant regions (in most cases the thirty largest) in a subset of images. Although the full database contains over 2000 images, the assessment set was restricted to 100 because of the length of time required to manually classify such segmented images. The classes used were chosen during the classification rather than beforehand so that no restriction could be placed upon the types given. A by-image examination content was performed and if a class did not exist to adequately describe what was seen, then a new class was selected. Once classification was complete, 17 distinct classes plus an unknown class for those not otherwise classified had been identified and over 1800 regions had been classified. When assigning a class to a region, it was sometimes not possible to make an exact classification, and in some cases it was necessary to apply more than one class to a single region. This situation sometimes arose either because of a segmentation error or simply because of the nature of the region in question. The resulting data set was not made available to the algorithm powering the query system and was only used in order to assess its performance. During the assessment of the system, the regions offered as query results were compared to the original key. Success was assumed if any of the selected region's classes matched with any of the original key region's classes, although a result region classed as unknown was not considered a success but was counted separately. A query result set comprised a set of regions, one from each of ten different images. These regions caused the highest activation of the RBF nodes indicating the closest matches to the search key.

In the first test, an automatic process was employed to use all classified regions as search keys in the initial query with the assumption of no a priori information and to analyse the resulting images. In some cases, the images returned were classed as unknown (untagged). The accuracy calculations have taken into account their presence. It was found that the `first guess' pass, 47.05% of classified regions offered, had a class matching that of the original key region.

In a second test, it was necessary to examine the effect of re-classification and the creation of complex RBF networks. In order to carry out a complete test, an automatic process was used to perform the evaluation. To replace the human-classification component, the tags provided in the manual classification process were used instead. Any regions offered which were unknown were not included in the test data sent to the LVQ algorithm. For each classified region in the data set, three re-classification processes were performed and the results recorded at the end of the last one. In this case, 76.60% of classified regions offered by the system matched with the class(es) of the originally selected region. Note that with a human operator classifying the result set, this result would improve, since during the automatic assessment, any unclassified regions were excluded from the LVQ training data. This would not be the case in a manual classification since it is likely that all regions would be tagged by the user which would improve the test data and thus potentially improve the results.

A sample run is shown in the figures. Figure  1 shows the region selected as the key for the query. Figure  2 shows the images returned after the first run and figure  3 shows those returned after four manual re-classification cycles. Note that the run which produced figure  3 only returned five images. This is because the system is satisfied that no other images in the database contain regions which satisfy the criteria of the search. A second example of the results of re-classification is shown in figure  4 . Here the region depicting the swimming pool in the image at top-left was given as the search key. After just two re-classification cycles, the image set shown in figure  4 resulted.

Although these tests were carried out on a small database sub-set, examination of manually guided queries on the full database leads us to believe that these results are representative of the overall situation.